John Philpin : Lifestream
Rebel Ridge, 2024
Watched on Saturday October 5, 2024.
Brought to you by Identity Woman and Infominer.
Support this collaboration on Patreon!!!
Watched on Saturday October 5, 2024.
こんにちは、富士榮です。
先日お知らせしたVittorio Bertocciアワードと並行してDIAF(Digital Identity Advancement Foundation)が提供する個人向けの世界2大アイデンティティ・アワードであるKim Cameronアワードの受賞者であるMatthew SpenceがIdentiverse 2024への参加レポートを書いています。
DIAFではアワード受賞者にIdentiverseやIIW、EICなどのカンファレンスへの参加をサポートしており、有能でやる気があっても費用面で課題がある若手などへのスポンサーをしています。
ダイバーシティを確保のためにはこのような取り組みは非常に有用ですね。日本でも何かやれないかなぁ、、、と思いますが、まずは日本からもDIAFのアワードにApplyしてみる方が出てくることに期待です。
Database Remote-Copy Tool For SQLite (draft)
Neat new SQLite utilities often show up in branches of the SQLite repository. Here's a new one from last month:sqlite3-rsync
, providing tools for efficiently creating and updating copies of WAL-mode SQLite databases on either the same machine or across remote machines via SSH.
The way it works is neat, inspired by rsync
(hence the tool's name):
The protocol is for the replica to send a cryptographic hash of each of its pages over to the origin side, then the origin sends back the complete content of any page for which the hash does not match.
SQLite's default page size is 4096 bytes and a hash is 20 bytes, so if nothing has changed then the client will transmit 0.5% of the database size in hashes and get nothing back in return.
The tool takes full advantage of SQLite's WAL mode - when you run it you'll get an exact snapshot of the database state as it existed at the moment the copy was initiated, even if the source database continues to apply changes.
I wrote up a TIL on how to compile it - short version:
cd /tmp
git clone https://github.com/sqlite/sqlite.git
cd sqlite
git checkout sqlite3-rsync
./configure
make sqlite3.c
cd tool
gcc -o sqlite3-rsync sqlite3-rsync.c ../sqlite3.c -DSQLITE_ENABLE_DBPAGE_VTAB
./sqlite3-rsync --help
Via lobste.rs
Hybrid full-text search and vector search with SQLite
As part of Alex’s work on his sqlite-vec SQLite extension - adding fast vector lookups to SQLite - he’s been investigating hybrid search, where search results from both vector similarity and traditional full-text search are combined together.The most promising approach looks to be Reciprocal Rank Fusion, which combines the top ranked items from both approaches. Here’s Alex’s SQL query:
-- the sqlite-vec KNN vector search results with vec_matches as ( select article_id, row_number() over (order by distance) as rank_number, distance from vec_articles where headline_embedding match lembed(:query) and k = :k ), -- the FTS5 search results fts_matches as ( select rowid, row_number() over (order by rank) as rank_number, rank as score from fts_articles where headline match :query limit :k ), -- combine FTS5 + vector search results with RRF final as ( select articles.id, articles.headline, vec_matches.rank_number as vec_rank, fts_matches.rank_number as fts_rank, -- RRF algorithm ( coalesce(1.0 / (:rrf_k + fts_matches.rank_number), 0.0) * :weight_fts + coalesce(1.0 / (:rrf_k + vec_matches.rank_number), 0.0) * :weight_vec ) as combined_rank, vec_matches.distance as vec_distance, fts_matches.score as fts_score from fts_matches full outer join vec_matches on vec_matches.article_id = fts_matches.rowid join articles on articles.rowid = coalesce(fts_matches.rowid, vec_matches.article_id) order by combined_rank desc ) select * from final;I’ve been puzzled in the past over how to best do that because the distance scores from vector similarity and the relevance scores from FTS are meaningless in comparison to each other. RRF doesn’t even attempt to compare them - it uses them purely for row_number()
ranking within each set and combines the results based on that.
Tags: embeddings, sql, vector-search, sqlite, search, alex-garcia, full-text-search, rag
Following my piece about reading the news on paper, I came across this post from Anrew Schmelyun:
"I recently purchased a dot matrix printer from eBay, and thought it would be a great excuse to have a custom "front page" printed out and ready for me each day. So, that's what I built!"
What a neat idea: he's called a few APIs (the New York Times, Reddit, Open-Meteo, and so on), installed it to run on a Raspberry Pi, and connected it to an old-school dot matrix printer to create a kind of Telex newspaper each morning,
I'd thought about doing this with an e-ink display, but honestly, why not just print it out?
I think I would want to pick some different news sources (the NYT is no longer my go-to) and leave out Reddit in favor of links that my contacts had shared on, say, Mastodon, but this is really fun. I might try and put together something similar, albeit with my existing laser printer rather than a dot matrix setup.
[Link]
More on Kris
I ’never’ do this - so take this as the exception that proves the rule. We all know that 🖇️ Kris K recently passed - on Maui
My friend Randall wrote to me (these extracts reproduced with his permission)
Such a sad loss of a hero whom I actually knew …. Sarah Teed and I had the privilege of a hanging w/ him and John Prine when we played the Castle theater many years ago and I opened for him at the MACC many times sharing moments with him and his wife Lisa.
Randall went on and shared some Kris wisdom …
”He believed that songwriting is a spiritual communion of mind, body, and soul, and he believed that William Blake was correct in asserting that anyone divinely ordered for spiritual communion but buries his talent will be pursued by sorrow and desperation through life and by shame and confusion for eternity.
” (Blake) is telling you that you’ll be miserable if you don’t do what you’re supposed to do," Kristofferson said in the Ken Burns' documentary ‘Country Music’.”
Cute. A good watch on a lazy sunny afternoon.
Anyone successfully — or not — following Threads or BlueSky in Microblog?
I think there’s more work to be done to explore print as a modern product to support great writing and journalism. Lots has been said about its death — but comparatively little about its potential to live on in new forms.
I think print has a lot of life left in it: particularly if we overcome the idea of preserving the exact form it’s taken in the past and consider what a more modern, reconsidered print product might look like.
There’s a lot to be said for reading on paper. One of my more recent indulgences has been a daily subscription to The Financial Times, which on weekdays is a sober paper that reports the news fairly objectively. On weekends it’s a different beast: in particular it includes a magazine pull-out called How to Spend It that is apparently aimed at the worst people on earth and is generally indistinguishable from satire.
The Financial Times has been publishing since 1888, but some endeavors are much newer. Speaking of indistinguishable from satire, I subscribed to The Onion’s print edition, now it has been bought from its private equity owner. It’s been fun seeing it adopt similar membership strategies to other, more “serious” publications. Most exciting among those is its resumed print edition, which is an old idea given a new spin:
“I think for the same reason that 18-year-old kids are buying Taylor Swift on vinyl,” Jordan LaFlure, The Onion’s executive editor also told the Times, “we can introduce those same kids to the notion that a print publication is a much richer way to consume media.”
It’s not obvious to me that a similar strategy couldn’t work for other publications — or even as a digest of independent publications that work together. Would I buy a subscription to a paper edition of independent journalism across various topics? Absolutely I would, and I don’t think I’m alone. Think of it as a lo-fi RSS reader or a retro Apple News: articles I care about from around the web in a form factor that looks more like The New Yorker (or The Onion).
This product could take several forms. It could combine an algorithmic component — here are the writers I care about — with a more human-driven curatorial component from editors who want to highlight interesting journalism from sources the reader might not have encountered yet. Or it could be a purely editorial product with no algorithmic component: one size fits all, for every reader. Or you could subscribe to personalized editions with different human editors who get a cut of subscriptions for putting it all together. (A monthly tech periodical organized by Casey Newton or Molly White? Take my money.)
Publications like ProPublica (my current employer) and The 19th (which I’ve worked for previously) produce content that is more long-form journalism than breaking news, which is highly suitable for reading in a collected periodical. They also make their content freely available via a Creative Commons license, meaning that, technically, anyone could put this together. But it would clearly be better in partnership with newsrooms, with revenue and subscriber information flowing back to them in exchange for letting their journalism be included.
This isn’t a traditional startup: it’s hard for me to see how this product would enjoy the rapid growth or high valuations which justify venture investment. But it’s potentially a really interesting small business. If the numbers work out, it could also potentially be a fascinating add-on product for a service like Medium. There’s user and market research to be done here, but it’s possible that the decline of legacy print products does not necessarily mean that new print products won’t be successful.
The act of reading on paper feels different to sitting in front of a screen. Maybe I’m getting old, but I like sitting at the dining room table, leafing through print. It is an old school product that is a little like vinyl, but it also feels like I’m using my brain a bit differently. I’d love to do more of it. In a world where everything is digital, maybe a thoughtfully curated print product could be exactly what we need to slow down and engage more deeply. Or maybe not, but I think it would be cool.
I’d love to hear what you think. Am I alone in preferring an offline, analogue, tactile reading experience? Is there something here, or is the future of media entirely, irrevocably digital?
Gemini 1.5 Flash-8B is now production ready
Gemini 1.5 Flash-8B is "a smaller and faster variant of 1.5 Flash" - and is now released to production, at half the price of the 1.5 Flash model.It's really, really cheap:
$0.0375 per 1 million input tokens on prompts <128K $0.15 per 1 million output tokens on prompts <128K $0.01 per 1 million input tokens on cached prompts <128KPrices are doubled for prompts longer than 128K.
I believe images are still charged at a flat rate of 258 tokens, which I think means a single non-cached image with Flash should cost 0.00097 cents - a number so tiny I'm doubting if I got the calculation right.
OpenAI's cheapest model remains GPT-4o mini, at $0.15/1M input - though that drops to half of that for reused prompt prefixes thanks to their new prompt caching feature (or by half if you use batches, though those can’t be combined with OpenAI prompt caching. Gemini also offer half-off for batched requests).
Anthropic's cheapest model is still Claude 3 Haiku at $0.25/M, though that drops to $0.03/M for cached tokens (if you configure them correctly).
I've released llm-gemini 0.2 with support for the new model:
llm install -U llm-gemini
llm keys set gemini
# Paste API key here
llm -m gemini-1.5-flash-8b-latest "say hi"
Via @OfficialLoganK
Tags: vision-llms, gemini, anthropic, openai, ai, llms, google, generative-ai, llm
At first, I struggled to understand why anyone would want to write this way. My dialogue with ChatGPT was frustratingly meandering, as though I were excavating an essay instead of crafting one. But, when I thought about the psychological experience of writing, I began to see the value of the tool. ChatGPT was not generating professional prose all at once, but it was providing starting points: interesting research ideas to explore; mediocre paragraphs that might, with sufficient editing, become usable. For all its inefficiencies, this indirect approach did feel easier than staring at a blank page; “talking” to the chatbot about the article was more fun than toiling in quiet isolation. In the long run, I wasn’t saving time: I still needed to look up facts and write sentences in my own voice. But my exchanges seemed to reduce the maximum mental effort demanded of me.
Tags: writing, generative-ai, chatgpt, ai, llms
Fifteenth in the News Commons series.
This semester’s Beyond the Web salon series for the Ostrom Workshop and Hamilton Lugar School at Indiana University is themed Think Globally, Eat Here—Small Solutions for Big Tech Problems. I will give the opening talk, about the News Commons (subject of fourteen prior posts here) at noon (Eastern) next Tuesday, October 10. If you’re in town, please attend in person. If not, join us by Zoom. Do that here.
Our plan is to prototype and prove locally what can apply globally for local news, starting with what Columbia Journalism Review called “news deserts” back in 2017—a label that has since caught on. There are many efforts toward seeding and watering these deserts, most prominently Press Forward, which is devoting $500 million to that challenge.
Bloomington is advantaged by not being one of those deserts, and instead having a talented pool of local journals, journalists, and organizations—including its legacy newspaper—all doing good work that could still be improved by putting to use some of the innovations I’ll be talking about, and by working together as a commons.
So join the conversation. I look forward to seeing you in the room or on the wall (because one whole wall is our Zoom screen).
Announcing FLUX1.1 [pro] and the BFL API
FLUX is the image generation model family from Black Forest Labs, a startup founded by members of the team that previously created Stable Diffusion.Released today, FLUX1.1 [pro] continues the general trend of AI models getting both better and more efficient:
FLUX1.1 [pro] provides six times faster generation than its predecessor FLUX.1 [pro] while also improving image quality, prompt adherence, and diversity.
Black Forest Labs appear to have settled on a potentially workable business model: their smallest, fastest model FLUX.1 [schnell] is Apache 2 licensed. The next step up is FLUX.1 [dev] which is open weights for non-commercial use only. The [pro] models are closed weights, made available exclusively through their API or partnerships with other API providers.
I tried the new 1.1 model out using black-forest-labs/flux-1.1-pro on Replicate just now. Here's my prompt:
Photograph of a Faberge egg representing the California coast. It should be decorated with ornate pelicans and sea lions and a humpback whale.
The FLUX models have a reputation for being really good at following complex prompts. In this case I wanted the sea lions to appear in the egg design rather than looking at the egg from the beach, but I imagine I could get better results if I continued to iterate on my prompt.
The FLUX models are also better at applying text than any other image models I've tried myself.
Via Hacker News
Tags: stable-diffusion, ai, generative-ai, replicate
Today, we cover:
Industry pulse. Microsoft won’t follow Amazon in 5-day RTO, Cloudflare auto-mitigates world-record DDoS attack, California nearly regulates AI companies, Revolut starts selling performance management framework as a product, and more.
Open source business model struggles: Wordpress. Automattic, creator of Wordpress, is being sued by one of the largest WordPress hosting providers. The conflict fits into a trend of billion-dollar companies struggling to effectively monetize open source, and are changing tactics to limit their competition and increase their revenue.
OpenAI: biggest-ever fundraise, as key people keep quitting. OpenAI raised a $6.6B in funding on a $157B valuation, making it the largest fundraiser ever. And yet, cofounders and executives continue to depart. This time it’s the CTO, Chief Research Officer, the VP of Research. Also, OpenAI will most likely finally become a for-profit.
Why executive recruiters ignore tech professionals. Connecting with executive recruiters is a helpful strategy for engineering leaders. Senior product director shares key reasons these outreaches fail, and what you can do to improve chances.
1. Industry pulse Microsoft won’t follow Amazon in 5-day RTOIn Seattle, the two largest tech employers are Microsoft and Amazon. Both companies are willing and able to match compensation of each other, so when Amazon announced two weeks ago that it’s mandating a strict 5-days-from-office policy from 1 Jan 2025, the question was whether Microsoft would follow.
The company has internally confirmed it will not, as per Business Insider, Microsoft EVP Scott Guthrie told employees that Microsoft has no plans to tighten remote work policies. This means Microsoft workers can expect to spend 2-3 days per week in the office as a hybrid arrangement. I expect experienced Amazon software engineers who are unhappy about their workplace’s 5-days-per-week policy, and who live within commuting distance of a Microsoft office, to start applying for open positions at Microsoft.
Cloudflare auto-mitigates world-record DDoS attackAsk HN: What happens to ".io" TLD after UK gives back the Chagos Islands?
This morning on the BBC: UK will give sovereignty of Chagos Islands to Mauritius. The Chagos Islands include the area that the UK calls the British Indian Ocean Territory. The .io ccTLD uses the ISO-3166 two-letter country code for that designation.As the owner of datasette.io the question of what happens to that ccTLD is suddenly very relevant to me.
This Hacker News conversation has some useful information. It sounds like there's a very real possibility that .io
could be deleted after a few years notice - it's happened before, for ccTLDs such as .zr
for Zaire (which renamed to Democratic Republic of the Congo in 1997, with .zr
withdrawn in 2001) and .cs for Czechoslovakia, withdrawn in 1995.
Could .io
change status to the same kind of TLD as .museum
, unaffiliated with any particular geography? The convention is for two letter TLDs to exactly match ISO country codes, so that may not be an option.
Tags: dns, domains, hacker-news
📸 A couple o' friends hanging at the bus stop.
Just listened to Gilbert’s 2024 release - he’s 78 for goodness sake. Not bad - but the classics are definitely not improved .. like the original 🔗 🎵 Nothing Rhymed .. always a place in my heart for this one. (I always liked his singles - but it was my sister that introduced me to the albums (my natural musical habitat.)
A Posterity Post
I had cause to add a comment on a post that featured this image.
It is funny - but I think it is funny for all the reasons that the commenters were not seeing ….
A university is to educate .. not train. If you are wondering about the difference, ask yourself if you prefer your children to attend sex education classes or sex training classes.
Sadly most institutions have moved to thinking of themselves as trainers .. so they are ‘workplace ready’. Witness a presentation I sat in just last week where someone from an educational facility was telling us how they are currently revamping their syllabus so that their students could learn how best to work with Ai (sic) … see Stephen Fry’s latest missive.
I stood up and asked her which University she thought the people making Ai had taught them what they know.
(Apparently .. that’s different).
As for philosophy .. oh and let’s throw in ethics, humanities, liberal arts, critical thinking, hell, even english .. are (should be) all on the rise in education, because it is very clear that we need them …badly.
I get the picture .. funny.
But the humor doesn’t lie in the kid having it wrong and dreaming what might be .. it’s in the dad having it wrong and thinking the future is anything like the past.
Just for kicks I signed up for a month of 🥐 🔗 ..
A buttery smooth app for cross posting to Bluesky, Mastodon, and Threads.
Big benefit for me is that I can add multiple accounts for the same service - useful as we start to push out The Future Found
AND
When I send I can choose which accounts it sends to.
Next job (request?) - as I publish to MicroBlog or LinkedIn or Facebook or …. have that post appear in Croissant ready for me to decide where I want to cross post. #Maybe.
I had a fun time live blogging OpenAI DevDay yesterday - I’ve now shared notes about the live blogging system I threw other in a hurry on the day (with assistance from Claude and GPT-4o). Now that the smoke has settled a little, here are my impressions from the event.
Compared to last year Prompt caching, aka the big price drop GPT-4o audio via the new WebSocket Realtime API Model distillation is fine-tuning made much easier Let’s build developer tools, not digital God Compared to last yearComparison with the first DevDay in November 2023 are unavoidable. That event was much more keynote-driven: just in the keynote OpenAI released GPT-4 vision, and Assistants, and GPTs, and GPT-4 Turbo (with a massive price drop), and their text-to-speech API. It felt more like a launch-focused product event than something explicitly for developers.
This year was different. Media weren’t invited, there was no livestream, Sam Altman didn’t present the opening keynote (he was interviewed at the end of the day instead) and the new features, while impressive, were not as abundant.
Several features were released in the last few months that could have been saved for DevDay: GPT-4o mini and the o1 model family are two examples. I’m personally happy that OpenAI are shipping features like that as they become ready rather than holding them back for an event.
I’m a bit surprised they didn’t talk about Whisper Turbo at the conference though, released just the day before - especially since that’s one of the few pieces of technology they release under an open source (MIT) license.
This was clearly intended as an event by developers, for developers. If you don’t build software on top of OpenAI’s platform there wasn’t much to catch your attention here.
As someone who does build software on top of OpenAI, there was a ton of valuable and interesting stuff.
Prompt caching, aka the big price dropI was hoping we might see a price drop, seeing as there’s an ongoing pricing war between Gemini, Anthropic and OpenAI. We got one in an interesting shape: a 50% discount on input tokens for prompts with a shared prefix.
This isn’t a new idea: both Google Gemini and Claude offer a form of prompt caching discount, if you configure them correctly and make smart decisions about when and how the cache should come into effect.
The difference here is that OpenAI apply the discount automatically:
API calls to supported models will automatically benefit from Prompt Caching on prompts longer than 1,024 tokens. The API caches the longest prefix of a prompt that has been previously computed, starting at 1,024 tokens and increasing in 128-token increments. If you reuse prompts with common prefixes, we will automatically apply the Prompt Caching discount without requiring you to make any changes to your API integration.
50% off repeated long prompts is a pretty significant price reduction!
Anthropic's Claude implementation saves more money: 90% off rather than 50% - but is significantly more work to put into play.
Gemini’s caching requires you to pay per hour to keep your cache warm which makes it extremely difficult to effectively build against in comparison to the other two.
It's worth noting that OpenAI are not the first company to offer automated caching discounts: DeepSeek have offered that through their API for a few months.
GPT-4o audio via the new WebSocket Realtime APIAbsolutely the biggest announcement of the conference: the new Realtime API is effectively the API version of ChatGPT advanced voice mode, a user-facing feature that finally rolled out to everyone just a week ago.
This means we can finally tap directly into GPT-4o’s multimodal audio support: we can send audio directly into the model (without first transcribing it to text via something like Whisper), and we can have it directly return speech without needing to run a separate text-to-speech model.
The way they chose to expose this is interesting: it’s not (yet) part of their existing chat completions API, instead using an entirely new API pattern built around WebSockets.
They designed it like that because they wanted it to be as realtime as possible: the API lets you constantly stream audio and text in both directions, and even supports allowing users to speak over and interrupt the model!
So far the Realtime API supports text, audio and function call / tool usage - but doesn't (yet) support image input (I've been assured that's coming soon). The combination of audio and function calling is super exciting alone though - several of the demos at DevDay used these to build fun voice-driven interactive web applications.
I like this WebSocket-focused API design a lot. My only hesitation is that, since an API key is needed to open a WebSocket connection, actually running this in production involves spinning up an authenticating WebSocket proxy. I hope OpenAI can provide a less code-intensive way of solving this in the future.
Code they showed during the event demonstrated using the native browser WebSocket
class directly, but I can't find those code examples online now. I hope they publish it soon. For the moment the best things to look at are the openai-realtime-api-beta and openai-realtime-console repositories.
The new playground/realtime debugging tool - the OpenAI playground for the Realtime API - is a lot of fun to try out too.
Model distillation is fine-tuning made much easierThe other big developer-facing announcements were around model distillation, which to be honest is more of a usability enhancement and minor rebranding of their existing fine-tuning features.
OpenAI have offered fine-tuning for a few years now, most recently against their GPT-4o and GPT-4o mini models. They’ve practically been begging people to try it out, offering generous free tiers in previous months:
Today [August 20th 2024] we’re launching fine-tuning for GPT-4o, one of the most requested features from developers. We are also offering 1M training tokens per day for free for every organization through September 23.
That free offer has now been extended. A footnote on the pricing page today:
Fine-tuning for GPT-4o and GPT-4o mini is free up to a daily token limit through October 31, 2024. For GPT-4o, each qualifying org gets up to 1M complimentary training tokens daily and any overage will be charged at the normal rate of $25.00/1M tokens. For GPT-4o mini, each qualifying org gets up to 2M complimentary training tokens daily and any overage will be charged at the normal rate of $3.00/1M tokens
The problem with fine-tuning is that it’s really hard to do effectively. I tried it a couple of years ago myself against GPT-3 - just to apply tags to my blog content - and got disappointing results which deterred me from spending more money iterating on the process.
To fine-tune a model effectively you need to gather a high quality set of examples and you need to construct a robust set of automated evaluations. These are some of the most challenging (and least well understood) problems in the whole nascent field of prompt engineering.
OpenAI’s solution is a bit of a rebrand. “Model distillation” is a form of fine-tuning where you effectively teach a smaller model how to do a task based on examples generated by a larger model. It’s a very effective technique. Meta recently boasted about how their impressive Llama 3.2 1B and 3B models were “taught” by their larger models:
[...] powerful teacher models can be leveraged to create smaller models that have improved performance. We used two methods—pruning and distillation—on the 1B and 3B models, making them the first highly capable lightweight Llama models that can fit on devices efficiently.
Yesterday OpenAI released two new features to help developers implement this pattern.
The first is stored completions. You can now pass a "store": true parameter to have OpenAI permanently store your prompt and its response in their backend, optionally with your own additional tags to help you filter the captured data later.
You can view your stored completions at platform.openai.com/chat-completions.
I’ve been doing effectively the same thing with my LLM command-line tool logging to a SQLite database for over a year now. It's a really productive pattern.
OpenAI pitch stored completions as a great way to collect a set of training data from their large models that you can later use to fine-tune (aka distill into) a smaller model.
The second, even more impactful feature, is evals. You can now define and run comprehensive prompt evaluations directly inside the OpenAI platform.
OpenAI’s new eval tool competes directly with a bunch of existing startups - I’m quite glad I didn’t invest much effort in this space myself!
The combination of evals and stored completions certainly seems like it should make the challenge of fine-tuning a custom model far more tractable.
The other fine-tuning announcement, greeted by applause in the room, was fine-tuning for images. This has always felt like one of the most obviously beneficial fine-tuning use-cases for me, since it’s much harder to get great image recognition results from sophisticated prompting alone.
From a strategic point of view this makes sense as well: it has become increasingly clear over the last year that many prompts are inherently transferable between models - it’s very easy to take an application with prompts designed for GPT-4o and switch it to Claude or Gemini or Llama with few if any changes required.
A fine-tuned model on the OpenAI platform is likely to be far more sticky.
Let’s build developer tools, not digital GodIn the last session of the day I furiously live blogged the Fireside Chat between Sam Altman and Kevin Weil, trying to capture as much of what they were saying as possible.
A bunch of the questions were about AGI. I’m personally quite uninterested in AGI: it’s always felt a bit too much like science fiction for me. I want useful AI-driven tools that help me solve the problems I want to solve.
One point of frustration: Sam referenced OpenAI’s five-level framework a few times. I found several news stories (many paywalled - here's one that isn't) about it but I can’t find a definitive URL on an OpenAI site that explains what it is! This is why you should always Give people something to link to so they can talk about your features and ideas.
Both Sam and Kevin seemed to be leaning away from AGI as a term. From my live blog notes (which paraphrase what was said unless I use quotation marks):
Sam says they're trying to avoid the term now because it has become so over-loaded. Instead they think about their new five steps framework.
"I feel a little bit less certain on that" with respect to the idea that an AGI will make a new scientific discovery.
Kevin: "There used to be this idea of AGI as a binary thing [...] I don't think that's how think about it any more".
Sam: Most people looking back in history won't agree when AGI happened. The turing test wooshed past and nobody cared.
I for one found this very reassuring. The thing I want from OpenAI is more of what we got yesterday: I want platform tools that I can build unique software on top of which I colud not have built previously.
If the ongoing, well-documented internal turmoil at OpenAI from the last year is a result of the organization reprioritizing towards shipping useful, reliable tools for developers (and consumers) over attempting to build a digital God, then I’m all for it.
And yet… OpenAI just this morning finalized a raise of another $6.5 billion dollars at a staggering $157 billion post-money valuation. That feels more like a digital God valuation to me than a platform for developers in an increasingly competitive space.
Tags: websockets, ai, openai, generative-ai, llms
"Addressing the Maker-Taker challenge is essential for the long-term sustainability of open source projects. Drupal's approach may provide a constructive solution not just for WordPress, but for other communities facing similar issues."
Dries lays out a constructive approach to crediting open source contributors. There's no stick here: just a series of what amount to promotion and status levels in return for making contributions like "code, documentation, mentorship, marketing, event organization" and so on.
I've certainly had to deal with the maker-taker problem too, although not at the magnitude that either Drupal or WordPress need to consider it. When I worked on Elgg, the open source ecosystem was relatively underdeveloped, and I don't remember it being much of a problem. In contrast, Known plugged into a significantly more advanced ecosystem. The solution Dries lays out makes a ton of sense to me, and I wish we'd done more along these lines in both cases.
[Link]
It’s a pretty low bar …
Ethical Applications of AI to Public Sector Problems
Jacob Kaplan-Moss developed this model a few years ago (before the generative AI rush) while working with public-sector startups and is publishing it now. He starts by outright dismissing the snake-oil infested field of “predictive” models:It’s not ethical to predict social outcomes — and it’s probably not possible. Nearly everyone claiming to be able to do this is lying: their algorithms do not, in fact, make predictions that are any better than guesswork. […] Organizations acting in the public good should avoid this area like the plague, and call bullshit on anyone making claims of an ability to predict social behavior.
Jacob then differentiates assistive AI and automated AI. Assistive AI helps human operators process and consume information, while leaving the human to take action on it. Automated AI acts upon that information without human oversight.
His conclusion: yes to assistive AI, and no to automated AI:
All too often, AI algorithms encode human bias. And in the public sector, failure carries real life or death consequences. In the private sector, companies can decide that a certain failure rate is OK and let the algorithm do its thing. But when citizens interact with their governments, they have an expectation of fairness, which, because AI judgement will always be available, it cannot offer.
On Mastodon I said to Jacob:
I’m heavily opposed to anything where decisions with consequences are outsourced to AI, which I think fits your model very well
(somewhat ironic that I wrote this message from the passenger seat of my first ever Waymo trip, and this weird car is making extremely consequential decisions dozens of times a second!)
Which sparked an interesting conversation about why life-or-death decisions made by self-driving cars feel different from decisions about social services. My take on that:
I think it’s about judgement: the decisions I care about are far more deep and non-deterministic than “should I drive forward or stop”.
Where there’s moral ambiguity, I want a human to own the decision both so there’s a chance for empathy, and also for someone to own the accountability for the choice.
That idea of ownership and accountability for decision making feels critical to me. A giant black box of matrix multiplication cannot take accountability for “decisions” that it makes.
Tags: jacob-kaplan-moss, ai, ethics
Building an automatically updating live blog in Django
Here's an extended write-up of how I implemented the live blog feature I used for my coverage of OpenAI DevDay yesterday. I built the first version using Claude while waiting for the keynote to start, then upgraded it during the lunch break with the help of GPT-4o to add sort options and incremental fetching of new updates.Tags: claude, ai, django, llms, javascript, ai-assisted-programming, generative-ai, chatgpt
So having a hard time connecting Threads to Micro.Blog.. meanwhile BlueSky connected - no problem .. but I can’t find the account to follow on Micro.Blog …
So far Mighty Masto is the only one doing both .. connect AND follow ..
Anyone else with similar challenges!
And once more from a different place.
🪦 Farewell Kris - taking your own last ride - but nothing like Casey’s.
"A couple months ago I was hanging out with my aunt, and she mentioned her cable+internet bill was around $250 per month. I thought that was insane and that I should do something about it. She's a 75 year old retiree that watches baseball and the hallmark channel, and she shouldn't have to pay as much as a car payment every month to do it."
What follows is a very smart way to share media profiles with a family member who doesn't live in your house, using Tailscale as a way to make them seamlessly appear like they're a part of your household.
Tailscale is easy-to-use and is virtually magic. I use it across my devices, and recommend it to others. This is a use case that makes a lot of sense.
[Link]
Published in 1992
Hope this gets fixed before the season starts …
A radio item
Over on my blog about infrastructure, I put up a brief post about WART, volunteer-powered community radio station with studios in a railroad caboose, that was lost in the flood that just devastated Marshall, North Carolina.
Write once, publish everywhere
Dave turned me on to Croissant today. Looks good. I’d even be willing to pay the monthly fee to post once across Bluesky, Mastodon, Threads, and Xitter. But it appears to be only for iOS mobile devices. I have some of those (including a new iPhone 16 Pro), but I mostly write on a computer. So I’ll hold out for real textcasting, like Tim Carmody talks up here. Because why should you have to post separately at all those places? Why should you have to go to a place at all, when you’ve got your own devices to write on and distribute from?
A heading convention
I started curating my photos (e.g. these) in the last millennium using this date-based naming convention: YYYY_MM_DD_topical-text_NNN.jpg (where the NNNs are just sequential numbers and the file type suffix could be .jpg, .arw, .cr2, .png or whatever. Same for folder titles.) So, because I don’t want a new title for every day I do this, I’m adopting the same convention, at least for now.
Not fast enough
In The End of Advertising, Michael Mignano says (in the subhead, and beyond), The business model that funded the internet is going away, and the open web will never be the same. He says AI is already killing it, by giving us answers to everything, and performing other handy tasks, without advertising to distract or annoy us. He also says AI services will attempt to invent ads, but that’s a losing proposition, mostly because it won’t work and we’ll hate it, but also because “content wants to be free.” (I submit that no art forms, ever, wanted to be called “content.”) I agree. I also agree that “Premium content will become even more premium.” He concludes, “the relationship between us and publishers will become much more transactional and direct. And we will feel it. Over time, it’ll be a new internet, and the open web will be a memory. Great content will still find a way to reach us, just like it always has. But we’ll look back on the first few decades of the internet as the golden age of content, when everything felt free.” Well, you’re reading some premium content right now, and it’s free. Thanks to what I do here, I can make money in other ways. We call those because effects.
🔗 Daring Fireball: ‘Meta and Apple: Same Game, Different Rules’
Not even sure it is the same game.
Men have no more time to understand anything. They buy things all ready made at the shops. But there is no shop anywhere where one can buy friendship, and so men have no friends any more.
💬 Antoine de Saint-Exupéry
He means all of us - different times - BUT - maybe it is more applicable to men? What say you?
You might not recognize the words - 🔗🎵 but you will certainly recognize the song. Superb re-imagination.
How did i get here?
Sitting on this spinning sphere
When will I leave here
We know death is always near
I try to understand but answers don’t come easily
Friends always deceiving me
Is this what life’s supposed to be?
Human animals
We roam the earth in search of blood
Hope for humanity?
I don’t think there ever was
We’re in perpetual war, and that’s the only law
Can I change it? I doubt it
So I write songs about it
I was young once but now I’ve grown old
Right in front of your еyes you’ve seen my life unfold
I had no choicе as I became the underdog’s voice
A young black kid… look what the fuck I did!
When I’m gone there’ll be someone to carry on
We can give in, give up, or we can stay strong
How accepting of the bullshit we’ve all become
This whole world is… comfortably numb
Hello… is anybody out there?
Can anybody hear me?
Hello… is anybody out there?
Can anybody hear me?
You’ve got a TV, a computer, so you don’t care
A roof, some clothes, some food
That’s right, it’s all there
Lock yourself in your house, try to forget about
The millions dyin' from wars, starvation and drought
You’ve got your own problems - yes, we all do
Nobody is free from stress, not him, or me, or you
But still we judge each other, we want to pick a fight
In the streets we kill each other, each and every weeknight
I don’t want no drama but I carry a gun
I don’t want no problems but they’ll probably come
So we get high to wash away the pain
Self-medication, but that shit is all done in vain
We think that we can hide away from the misery
You can close your eyes but still can plainly see
There’s nowhere to run…
You’re just comfortably numb
Hello… is anybody out there?
Can anybody hear me?
Hello… is anybody out there?
Can anybody hear me?
I spent half my life just trying to survive
And the other half trying to keep you all alive
For some reason I still care
I’ve done my dirt
It’s amazing that i’m still here
There’s true evil out there, I’ve been trying to warn you
You could meet the devil himself, then what you gonna do?
No time to pray, no chance to run
Listen… ‘cause I’ve been on both sides of the gun
As you stand before me we’re all here as one
We gotta come together or our chances are none
Maybe I’m just a dreamer, too many obstacles
The though of humans coexisting, that’s impossible
Better to stay numb, live in denial
We’re all going to hell and we did it to ourselves
Too much religious beef, too much racist shit
And if there is a God we probably make him sick
Hello… is anybody out there?
Can anybody hear me?
Hello… is anybody out there?
Can anybody hear me?
I'm at OpenAI DevDay in San Francisco, and I'm trying something new: a live blog, where this entry will be updated with new notes during the event.
See OpenAI DevDay: Let’s build developer tools, not digital God for my notes written after the event, and Building an automatically updating live blog in Django for details about how this live blogging system worked under the hood.
Tags: blogging, ai, openai, generative-ai, llms
Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:
Before we start: The Pragmatic Engineer Podcast started off last week, with episode #1: AI tools for software engineers, but without the hype. New episodes come every other Wednesday. Thank you to everyone who listened to this first one. If you enjoy podcasts, please do add it on Apple, Spotify, YouTube or your in your favorite player – and you will have episodes automatically show up, while also helping the show.
How do you deal with bugs in software products you build? This topic seems very under-discussed, but it affects pretty much every software team. To find out what professionals think, with we reached out to two dozen engineering leaders and software engineers, who kindly shared approaches which work for their team and organization.
This topic is evergreen, and one that has outsized (potentially decisive!) impact on product quality and user experience.
In this issue, we cover:
Finding bugs. Dogfood products where possible, invest in test automation, stay close to users, and more.
Reporting bugs. Making it easy for users to report bugs, having an accessible engineering team, and scaling the reporting process by team or company size.
Bug triaging. Figure out which bugs are critical by using bug priority rubrics, consider priority definitions, and get inspired by Spotify’s bug prioritization framework.
Don’t manage bugs: fix them! See bug fixing as like gardening, not maintenance. Consider approaches like deleting all amassed bugs, and regularly pruning the backlog.
Zero bugs policy. An approach where all inbound bugs are either fixed on the spot: or the bug is deemed invalid. A subset of this approach is to fix all bugs for recently released features – which is what Figma does.
As a refresher, we have a few previous deepdives that are related to this topic:
Incident review and postmortem best practices
How Big Tech does Quality Assurance (QA)
QA approaches across the industry
Thank you to everyone who contributed insights to this article: Ahmed Saher (engineering manager), Anaïs van Asselt (senior QA engineer), Andrea Sipos (product leader), Bernd Kampl (Software Engineering Team Lead), Jason Diller (VP of Engineering), John Cutler (product leader), Magnus L. Udbjørg (CTO), Michał Borek (Principal Engineer), Rebecca Frost (QA leader), Rebecca Holm Ring (engineering leader), Ruben Weijers (engineering manager), Ryan Hanni (Director of Engineering), Serdar Biyik (engineering manager), Walter de Bruijn (Head of Engineering Productivity)
1. Finding bugsHow can we be confident that the software we release has no known issues? We need to validate that it works correctly and there are common approaches for this.
Dogfood products. The term “dogfooding” is the name of the common practice of devs and employees using a product while they are building it, pre-release. For example, when I worked at Uber, the company issued free credits for staff to use the internal beta app for rides and food deliveries. At Skype, we used running internal beta versions of Skype for all internal chat and video calling. The business gave Skype credits to employees, so we could dogfood paid features like landline calls. Spotify does the same, as Rebecca Holm Ring, a former engineering manager there, shares:
“Each employee at Spotify is expected to be on the master or internal Spotify release, and report any issues they experience. A problem here though is that most Spotify employees will be iOS users, and so the Android app doesn’t get nearly enough testing before rolling out.”
Invest in test automation. Anaïs van Asselt – senior QA at Choco – shares their approach:
“Proactively finding and fixing bugs before they reach production is crucial. We invest in various test automation techniques to act as quality gates. Additionally, we practice bug hunting and gradually roll out new features to a limited user base, allowing us to catch and fix bugs early, reducing their overall cost.”
At smaller companies, be close to users. These places tend to be closer to users and can use this to build a relationship with users who get invested in the product and the reporting of bugs. Bernd Kampl – Software Engineering Team Lead at Anyline, a smaller cartech AI company – shares:
“As we are still smaller, we usually find bugs when our customers report them. If that's not the case, we identify them mostly during day-to-day development work.”
Magnus Udbjørg is CTO of Testaviva, a 50-person startup in Denmark. His take is that it’s optimal to build trust with users so they report issues:
“To find, identify, and prevent bugs, we foster a culture of dialogue and mutual understanding. We believe our developers need to understand our users deeply, and sometimes, our users need to understand the technical constraints we face. Open communication is key to building trust.”
A fair question is why not do lots of testing, themselves? The smaller the company and the fewer the customers, the more it feels too expensive to invest a lot in testing, early on. Of course, there are always countercases, like how Figma spent nearly 3 years iterating on the first release, in order to get the performance of their collaborative, web-based editor right, to give users a fun “wow moment.” Worth noting that Figma is a product the dev team used continuously while developing it, getting lots of testing during the building phase. We cover Figma’s engineering culture in a deep dive.
Consider alpha and beta testing at larger companies. Alpha and beta testing is about giving customers access to unfinished, less stable versions of a product. “Alpha” usually refers to a latest build that has had little to no QA testing. “Beta” versions have had some testing, but not as much as a full release.
Rebecca Holm Ring shares how this worked at Spotify:
“There’s an Alpha testing, and Beta testing program, where external users are on the master branch and release branch respectively, and are expected to report any issues they encounter. With these versions of Spotify, it is also possible for a user to report a bug directly from the app, and logs will be attached as the bug report will be automatically added in JIRA.”
Automation: testing and code analysis. Unit tests, integration tests, end-to-end-tests, and other automated tests, are great ways to catch regressions, which is a software bug introduced into a feature after the feature was working correctly; the feature has ‘regressed’ into a faulty state.
This is true for static code analysis and other tools that automate quality assurance. We cover more on these methods in Shipping to production and QA approaches across the industry.
Code reviews. These serve multiple purposes, offering a second pair of eyes to double check code, spread knowledge, and follow not-yet-automated conventions, and more. Catching bugs before they make it into the codebase is an occasional side effect.
Even so, bugs can easily slip through code reviews, which are nowhere near a perfect way to defend against shipping bugs and regressions.
Define what a bug is. Users often report “bugs” when they mean missing features, so it can be helpful for teams to agree what a bug is and how to categorize them. In general, a bug is a flaw that results in a software product behaving incorrectly. Categorizations can be granular, like splitting bugs into concurrency bugs, syntax ones, arithmetic, logical errors, human errors and so on.
The simplest categorization is to split bugs into functional ones, when the behavior of the software is clearly wrong, and non-functional ones, when a bug is revealed in things like a system slowing down, increased latency, and other harder-to-spot issues.
It might be helpful to devise your own categorizations, based on the type of bugs you observe, and in a way that’s helpful for your product and organization.
2. Users reporting bugsGathering bugs can be a great source of data, providing a sense of product quality for feedback to teams, the organization, or company. However, data quality depends on how good the bug reporting process is – and how likely people are to report bugs!
Great reports and data come from simple, suitable processes. Features of useful bug reports:
Useful metadata (e.g. version, device, system metrics)
Relevant context (e.g. on mobile while connected to bluetooth speaker and poor connectivity, on a server in this region during lunch hour, on a debug build with these feature flags active, etc)
Straightforward to reproduce, or have reproduction steps
Reported by users who trust a reported bug will be fixed
Bad reports can create extra work and poor bug reporting processes can cause people to not commit to recording issues in the first place, and a spiral is created of deteriorating product quality, with the engineering team clueless of how bad things are.
To avoid an outcome like that, here are some processes tech companies use to support good bug reporting processes.
Make it easy to create quality bug reports. Walter de Bruijn, Head of Engineering Productivity at Miro suggests this is critical:
“The easier the reporting is, the more you will learn. For internally discovered bugs an internal #bugs Slack channel can go a long way.
One of my best recommendations is that you start logging bugs properly: you can’t manage what you can’t see. You need a good, well-documented, and known process for reporting bugs and a follow-up.”
QA leader Rebecca Frost on why quality bug reports count:
“Capture meaningful metadata on your bugs now to help with insights later.
Make sure your funnel for incoming bugs is helping not hurting you. If bugs are coming in from all over and the quality of the info on the bugs is a mess, consider routing bug reports through one team or group.”
Make the reporting process accessible. If creating a bug report is too complicated, it discourages reporting. There are ways to make it accessible:
Don’t make less technical users use unfamiliar tech jargon
Make it easy to find version numbers and other data required for bug reports. Even better, collect it automatically
Consider having engineers help support teams to know which questions to ask to get the right data for reports
A example of effective engineering support is at SF-based scaleup Ontra, as shared by director of engineering, Ryan Hanni:
Scaling bug reporting processes“We have an Engineering Support team that sits between Product Support/Customer Success and R&D, which helps bridge the customer with R&D teams potentially further away from that context.”
There’s no one process that works best everywhere. Here are some common approaches by company size:
Smaller companies and startups: bug reports are usually simple, and the reporting process is lean because time is precious and knowledge is dense. Such workplaces are small enough that most tech folks can keep tabs on what’s happening, and people can submit bug reports pretty easily.
There’s rarely a need for formal processes. Here are some efficient, less formal ones:
Set up a #bug-reports channel on Slack/Teams or other chat product
Use a #bugs tag in the ticket system of choice to keep track of issues
Prefix bug fix pull requests or commits with [Bugfix]
… and similar, simple but effective conventions that anyone can start using
Mid-sized companies and scaleups: process matters more, and these places are big enough for it to be wasteful for everyone to keep tabs on reported bugs. There are also more bug reports, and it’s a time waster to get repeated information and metadata in bug reports.
Bug report templates and processes also matter. Good onboarding and documentation for processes and standards for bugs can have a big impact on efficiency.
Large companies: investing in automated processes is worthwhile due to the size and nature of the business:
Noisy bug reporting means lots of duplicate reports
“JIRA ping pong”: tickets created in JIRA are continually reassigned between teams and end up ‘returned to sender’
Time wasted discussing bugs: lots of overhead on debating whether a bug is a feature request, who’s best placed to fix it, etc
Here’s a good example of what JIRA ping pong looks like. Engineering manager Rebecca Holm Ring shares how it plays out a larger company:
“Bugs showing up in one user-facing feature could be caused by a number of different teams that own the deeper layers of the tech stack. Still, reports would always first land on the user facing team!
There would often be a game of ping pong, where bugs got sent back-and-forth between different teams, and sometimes found the right owners. In turn, these owners didn’t feel it was a high enough issue for them to fix. However, for the user facing team, it would often be a higher priority to get it resolved. As many different teams were involved in building the Spotify app, conflicting priorities often made bugs linger longer than they should have, and occasionally required intervention from the higher-ups to get these bugs prioritized by the owning team.”
At larger companies, some things can help deal with an ever-growing pile of bug reports, and improve processes and tooling:
Automation across the bug reporting process
Automated data collection for gathering context
Bug reporting templates for internal use
Automatic code/team attribution for inbound reports
Ryan Hanni, director of engineering at Ontra, shares examples of manual and mostly-automated processes he’s implemented at different stages of an org’s lifecycle:
Manual process:
Use a bug template in Trello, JIRA, or similar
Notify Eng Support and copy the bug description to their slack channel
→ Eng support aids in determining severity, and the team takes it from there
→ Eng Director uses a Google Spreadsheet to track all bugs, and includes details like report date and links
→ Eng Director does monthly follow ups on bugs, and adds details to the spreadsheet, like the timestamp of a fix. This spreadsheet is used for calculating stats like time-to-resolution for DORA metrics.
Partially Automated:
Use a bug template in Trello, JIRA, or similar.
With a press of a button, automatically post to Slack
→ Eng Support 'triages' the bug with the dev team in a thread
→ For high or medium severity bugs, support copies the details to an Airtable form and submits it
→ The form submitted uses automation to:
Post to Slack
Send an email to internal google groups for stakeholders and departments
Automatically adds bug information into a Airtable columns
→ Partially automated reporting:
Metrics for ‘Reported at’ date and severity are automatically filled in
Metrics are filed on a monthly basis to generate stats like DORA metrics and cycle time to fix (Time to Respond, Time to Fix, Total Time In Prod, etc).
→ When a bug is fixed, teams reply to the automated email detailing when it will be released to customers
Mostly Automated:
Use a bug template in Trello, JIRA, or similar.
→Use automation to send a message to Eng Support slack.
→ Triage happens with the Dev Team and Eng Support
→ Work moves through Jira columns (Kanban style), into a Done column
→ A release to production happens and the Jira card gets an automatic “released at” date
→ After a release to Prod, Jira sends a slack/email message with issue details to close the loop for a resolved bug of a given severity.
→ Reporting occurs by generating reports in JIRA. Run reports with all relevant metrics to the org to get information like cycle time, time to fix, response time, etc
→ Visuals for share outs on the bug fixing process is manually generated in Google Sheets
3. Bug triageYou now have a process for the bug reports to flow in, so the next step is to figure out which ones are critical, which are duplicates, and which ones not to bother with. Here are some common approaches:
I’m thrilled to report that the OpenID Connect specifications have now been published as ISO/IEC standards. They are:
ISO/IEC 26131:2024 — Information technology — OpenID connect — OpenID connect core 1.0 incorporating errata set 2 ISO/IEC 26132:2024 — Information technology — OpenID connect — OpenID connect discovery 1.0 incorporating errata set 2 ISO/IEC 26133:2024 — Information technology — OpenID connect — OpenID connect dynamic client registration 1.0 incorporating errata set 2 ISO/IEC 26134:2024 — Information technology — OpenID connect — OpenID connect RP-initiated logout 1.0 ISO/IEC 26135:2024 — Information technology — OpenID connect — OpenID connect session management 1.0 ISO/IEC 26136:2024 — Information technology — OpenID connect — OpenID connect front-channel logout 1.0 ISO/IEC 26137:2024 — Information technology — OpenID connect — OpenID connect back-channel logout 1.0 incorporating errata set 1 ISO/IEC 26138:2024 — Information technology — OpenID connect — OAuth 2.0 multiple response type encoding practices ISO/IEC 26139:2024 — Information technology — OpenID connect — OAuth 2.0 form post response modeI submitted the OpenID Connect specifications for publication by ISO as Publicly Available Specifications (PAS) for the OpenID Foundation in December 2023. Following the ISO approval vote, they are now published. This should foster even broader adoption of OpenID Connect by enabling deployments in jurisdictions around the world that have legal requirements to use specifications from standards bodies recognized by international treaties, of which ISO is one.
Before submitting the specifications, the OpenID Connect working group diligently worked through the process of applying errata corrections to the specifications, so that the ISO versions would have all known corrections incorporated.
Having successfully gone through the ISO PAS submission process once, the OpenID Foundation now plans to submit additional families of final specifications for publication by ISO. These include the FAPI 1.0 specifications, and once they’re final, the eKYC-IDA specifications and FAPI 2.0 specifications.
Thanks to all who helped us achieve this significant accomplishment!
Looks like they got an early start on the releases, with the first new Whisper model since November 2023.
Whisper Turbo is a new speech-to-text model that fits the continued trend of distilled models getting smaller and faster while maintaining the same quality as larger models.
large-v3-turbo
is 809M parameters - slightly larger than the 769M medium but significantly smaller than the 1550M large. OpenAI claim its 8x faster than large and requires 6GB of VRAM compared to 10GB for the larger model.
The model file is a 1.6GB download. OpenAI continue to make Whisper (both code and model weights) available under the MIT license.
It’s already supported in both Hugging Face transformers - live demo here - and in mlx-whisper on Apple Silicon, via Awni Hannun:
import mlx_whisper
print(mlx_whisper.transcribe(
"path/to/audio",
path_or_hf_repo="mlx-community/whisper-turbo"
)["text"])
Awni reports:
Transcribes 12 minutes in 14 seconds on an M2 Ultra (~50X faster than real time).
Would a blog be a blog if it went behind a paywall, or if you needed a subscription to read it?
Of course not. Blogs are on the open Web, and tend to stay there so long as they don’t move away from their original location.
Same should go for podcasts. “Wherever you get your podcasts” certifies the open nature of podcasting.
But now the New York Times is putting Ezra Klein’s podcast archives behind a paywall.
Never mind how icky this is on several grounds. Our challenge now is classification. We need a new noun for restricted ‘casts such as Ezra’s. I suggest wallcasts.
For subscription-only ‘casts, such as some on SiriusXM*, I suggest paycasts.
Bottom line: It can’t be a podcast if you have to pay for any of it, including archives.
By the way, it won’t matter if a Times subscription opens wallcast archives, as it does for print. By putting their podcast archives behind a paywall, the Times is changing the DNA of those casts. A wallcast is not a podcast. Full stop.
Spread the words.
*SiriusXM’s paycasts include “SmartLess,” “Freakonomics Radio,” “The Joel Osteen Podcast,” “Last Podcast on the Left,” and “Andy Cohen’s Daddy Diaries.” They require a subscription to SiriusXM or its Podcasts+ service. Some, such as “Marvel’s Wastelanders” and “Marvel/Method also require a subscription. I’m not sure what kind. (FWIW, I’ve been a SiriusXM subscriber since 2005, but only listen to live subscription streams. I’ve never listened to any of its podcasts.) SiriusXM does have some shows in podcast form, however. Examples are “The Megyn Kelly Show,” “Best Friends with Nicole Byer and Sasheer Zamata,” and “Chasing Life with Dr. Sanjay Gupta.” I believe it also has some wallcasts. For example, “SmartLess” episodes are on the open Web, but early access and bonus episodes are behind a paywall. Or so it seems to me in the here and now. I invite corrections.
I played around just a little bit with Google’s NotebookLM. It’s a Retrieval-Augmented Generation (RAG) tool. One thing it does is convert your content into a podcast format and it provides an impressive-sounding result. It's a little superficial on the content synthesis, but it is a podcast after all. Its other features are your typical RAG chat interface. All I can say is give it a try, or check out my example effort.
Here is a link to a single source document I provided it- Self Sovereign Identity A Systematic Review Mapping and Taxonomy and the link to the SSI Paper Podcast results.
Average score for an average movie. Casey? Never been overly bothered, but Matt is capable of so much more.
This review may contain spoilers.
Wolfs … not Wolves. Not sure why. Need to look it up. Other than that .. all good. They have a good relationship IRL … it comes through in the movie.
[Reddit is] mostly ported over entirely to Lit now. There are a few straggling pages that we're still working on, but most of what everyday typical users see and use is now entirely Lit based. This includes both logged out and logged in experiences.
Tags: web-components, reddit, lit-html, javascript
こんにちは、富士榮です。
しばらく別のネタばかりでSIDI Hubについてかけていませんでしたが、10月に入ったので東京開催秒読みということでベルリンレポートの続きを読んでいきます。
Users of a Trust Framework Analysis Tool
A major output of the SIDI Hub 2024 strategy, led by the Open Identity Exchange (OIX), will be a Trust Framework Comparison Tool. This will be bolstered by further analysis and normalization of Trust Frameworks supported by SIDI Hub. At the SIDI Berlin session, breakout groups shaped the value proposition and requirements for such a comparison tool, which will directly influence the final deliverable. Further information is found in the Rapporteur’s notes (next section).
信頼フレームワーク分析ツールのユーザー
Open Identity Exchange (OIX) が主導する SIDI Hub 2024 戦略の主な成果のひとつは、信頼フレームワーク比較ツールです。これは、SIDI Hub がサポートする信頼フレームワークのさらなる分析と標準化によって強化されます。SIDI Berlin セッションでは、分科会がこのような比較ツールの価値提案と要件を策定し、最終成果物に直接影響を与えることになります。詳細は、ラポータのメモ(次項)をご覧ください。
トラストフレームワークのマッピングに関して書かれていますね。
現在、各国で制定が進んでいるトラストフレームワークの相互運用が可能な状態にならないと国の間で相互運用性の担保ができなくなるのでここでいうマッピングは非常に重要です。OpenIDファウンデーションジャパンではOIXに協力する形で日本のトラストフレームワークのマッピングを支援しています。
BOLTS: Business, Operational, Legal, Technical, and Social
Given the above take-aways, which span Business, Operational, Legal, Technical, and Social forces that impact the global interoperability effort, the group will use a “BOLTS” framework as a core part of its Champion Use Case analysis.
BOLTS:ビジネス、運用、法律、技術、社会
グローバルな相互運用性への取り組みに影響を与えるビジネス、運用、法律、技術、社会の各分野における上記の要点を踏まえ、当グループは「BOLTS」フレームワークをチャンピオンユースケース分析の中核として使用します。
相互運用性を考える上では技術だけを考えていたは不十分です。ここにあるようにビジネス、運用、法律、社会を含めて考える必要がある、ということです。
Government Participation
A final point of reflection relates to the audience for SIDI Hub events. Given the light attendance from government officials in Berlin, the agenda skewed towards a technical audience that discussed technical possibilities. This is not ideal.
政府の参加
最後に、SIDI Hubのイベントの聴衆について考察したいと思います。ベルリンでの政府関係者の出席が少なかったため、技術的な可能性について議論する技術的な聴衆に偏ったアジェンダとなりました。これは理想的ではありません。
先に記載した通り、法律や社会についても検討が必要です。ベルリンでは政府機関の設備で開催したにもかかわらず確かにあまり多くの政府関係者が参加したわけではありませんでした。この辺りは日本開催をする際のバランスに関する考慮点となるでしょう。
SIDI Hub was founded to unite global audiences to define the users, benefits, and overall business case for globally interoperable digital identity to normalize approaches and define minimum requirements. It was, therefore, somewhat premature to attempt a solution-oriented agenda. With that said, the lessons were valuable, and SIDI Hub has had valuable contributions from European stakeholders through other avenues, e.g., the SIDI Paris Summit, eIDAS 2.0 documentation, etc. Regardless, the SIDI organizers have determined that baseline government participation will be a critical go/no-go criterion for the events planned in Washington, D.C., Tokyo, and Brazil.
SIDI Hubは、世界中のオーディエンスをまとめ、世界規模で相互運用可能なデジタルIDのユーザー、利点、全体的なビジネスケースを定義し、アプローチを標準化し、最低限の要件を定義するために設立されました。そのため、ソリューション志向のアジェンダを試みるには時期尚早でした。とはいえ、そこから得られた教訓は貴重であり、SIDIハブは、SIDIパリサミットやeIDAS 2.0文書など、他の手段を通じて欧州の利害関係者から貴重な貢献を得ることができました。それでも、SIDIの主催者は、ワシントンD.C.、東京、ブラジルで計画されているイベントについては、政府の基本的な参加が実施の可否を決定する重要な基準となると判断しました。
ベルリンでもユースケースの取りまとめ要件整備を行いました。次のワシントンDCや東京・ブラジルでの開催に向けて議論をしていく必要がありそうです。なお、ここに記載がある通りソリューションとして自立させるためのきっかけには早すぎるイメージはありました。しかし読者の皆さんは気にせずにアプライしてくださいね。
Helene was Western North Carolina‘s Katrina—especially for the counties surrounding Asheville: Buncombe, Mitchell, Henderson, McDowell, Rutherford, Haywood, Yancey, Burke, and some adjacent ones in North Carolina and Tennessee. As with Katrina, the issue wasn’t wind. It was flooding, especially along creeks and rivers. Most notably destructive was the French Broad River, which runs through Asheville. Hundreds of people are among the missing. Countless roads, including interstate and other major highways, are out. Towns and communities—Spruce Pine, Swananoa, Chimney Rock, Mitchell, Lake Lure, and many others—have been wiped away, or are in ruins. Roads across the region are gone, or closed. Electric, water, gas, sewer, and other utilities are expected to be down in many places for weeks.
One public utility that is working enough for people to keep up with news is radio. Many (perhaps most) stations are off the air, but some survive, and are providing constant service to residents and people out of the area who want to stay informed. The two I recommend most are Blue Ridge Public Radio (WCQS/88.1) and any of the local iHeart stations . All of the iHeart stations listed in the image above are carrying the same continuous live coverage, which is excellent. (I’m listening right now to the WWNC/570 stream.)
Of course, there’s lots of information on social media (e.g. BlueSky, Xitter, Threads), but if you want live coverage, radio still does what only it can do. Yes, you need special non-phone equipment to get it when the cell system doesn’t work, but a lot of us still have those things. Enjoy the medium while we still have it.
Item: WWNC just reported that WART/95.5 FM in Marshall, with its studios in a train caboose by the river, is gone (perhaps along with much of the town). [Later…For more on that, go here.]
More sources:
WISE/1310 stream WTMT/105.9 streamThis is cross-posted on Trunkli, my blog on infrastructure.
"I installed a box high up on a pole somewhere in the Mission of San Francisco. Inside is a crappy Android phone, set to Shazam constantly, 24 hours a day, 7 days a week. It's solar powered, and the mic is pointed down at the street below."
As surveillance goes, I'm into it. I appreciate the commentary:
"Heard of Shot Spotter? Microphones are installed across cities across the United States by police to detect gunshots, purported to not be very accurate. This is that, but for music."
I don't give it much time before someone figures out where it is and tries to mess with it, though.
[Link]
Some details on how it works from Riley on Twitter:
The phone has a Tasker script running on loop (even if the battery dies, it’ll restart when it boots again)
Script records 10 min of audio in airplane mode, then comes out of airplane mode and connects to nearby free WiFi.
Then uploads the audio file to my server, which splits it into 15 sec chunks that slightly overlap. Passes each to Shazam’s API (not public, but someone reverse engineered it and made a great Python package). Phone only uses 2% of power every hour when it’s not charging!
Via Kellan
Tags: hardware-hacking, music, android
I listened to the whole 15-minute podcast this morning. It was, indeed, surprisingly effective. It remains somewhere in the uncanny valley, but not at all in a creepy way. Just more in a “this is a bit vapid and phony” way. [...] But ultimately the conversation has all the flavor of a bowl of unseasoned white rice.
Tags: llms, generative-ai, notebooklm, ai, john-gruber, podcasts
In How and why to write letters to voters I discussed Vote Forward, my favorite way for those of us who aren’t in swing states to reach out to voters in swing states. The site works really well for adopting batches of voters, and downloading packets of form letters. As I close in on 1000 letters, though, I’m finding it isn’t great for tracking progress at scale. Here’s how my dashboard page looks.
With 50 bundles in play, many of which are farmed out to friends and neighbors who are helping with the project, it’s become cumbersome to keep track of which bundles are prepped (ready to mail) or not. Here is the checklist I needed to see.
VoteForward Dashboard Report mmorg: 1-UNPREPPED r23Pp: 2-UNPREPPED v9Kbo: 3-UNPREPPED wLMPw: 4-UNPREPPED 24L4o: 5-PREPPED 4nNnj: 6-PREPPED 5rQmV: 7-PREPPED ... YV4dL: 48-PREPPED zKjne: 49-PREPPED ZrKJz: 50-PREPPEDIf you’re in the same boat, here’s a piece of code you can use to make your own checklist. It’s gnarly, if you aren’t a programmer I advise you not even to look at it, just copy it, and then paste it into your browser to have it open a new window with your report.
Vote Forward checklist maker (expand to copy) javascript:(function(){ // First part: Adjust height of divs with inline styles document.querySelectorAll('div[style]').forEach(div => { let inlineStyle = div.getAttribute('style'); if (inlineStyle.includes('position: relative')) { div.style.height = '20000px'; // Set the height to 20000px } }); // Introduce a delay before processing the list of items setTimeout(() => { const items = document.querySelectorAll('li.bundle-list-item.individual'); let dataList = []; // Iterate over the items to capture data-testid and ID items.forEach(item => { let dataTestId = item.getAttribute('data-testid'); // Use the id attribute of the input element to extract the ID const toggleInput = item.querySelector('input.slide-out-toggle'); const toggleId = toggleInput ? toggleInput.getAttribute('id') : ''; // Extract the ID part from the toggleId pattern "toggle-24L4o-PREPPED" const id = toggleId ? toggleId.split('-')[1] : 'ID not found'; // Remove "bundle-" and the number part from dataTestId, keeping only "PREPPED" or "UNPREPPED" dataTestId = dataTestId.split('-').pop(); // Extract only the "PREPPED" or "UNPREPPED" part // Push the data into the array dataList.push({ dataTestId, id }); }); // Sort first by whether it's PREPPED or UNPREPPED (descending for UNPREPPED first), // then by the ID within each group dataList.sort((a, b) => { if (a.dataTestId.includes("PREPPED") && b.dataTestId.includes("UNPREPPED")) { return 1; // UNPREPPED comes before PREPPED } else if (a.dataTestId.includes("UNPREPPED") && b.dataTestId.includes("PREPPED")) { return -1; } // Sort by ID if they belong to the same category return a.id.localeCompare(b.id); }); // Prepare the output string let output = ''; dataList.forEach((item, index) => { output += `${item.id}: ${index + 1}-${item.dataTestId}\n`; }); // Open a new window with the output in a text area for easy copying let newWindow = window.open('', '', 'width=500,height=500'); newWindow.document.write('<html><body><h2>VoteForward Dashboard Report</h2><pre>' + output + '</pre></body></html>'); newWindow.document.close(); }, 2000); // Adjust delay as needed })();Here are instructions for Chrome/Edge, Safari, and Firefox. You might need to tell your browser to allow the popup window in which it writes the report.
Chrome/Edge: Open the VoteForward dashboard in your browser. Open the developer console: Windows/Linux: PressCtrl + Shift + J
.
Mac: Press Cmd + Option + J
.
Paste the code into the console.
Press Enter
to run the code.
Firefox:
Open the VoteForward dashboard in your browser.
Open the developer console:
Windows/Linux: Press Ctrl + Shift + K
.
Mac: Press Cmd + Option + K
.
Paste the code into the console.
Press Enter
to run the code.
Safari:
Open the VoteForward dashboard in your browser.
Enable the developer console (if it’s not already enabled):
Go to Safari > Preferences.
Click the Advanced tab.
Check “Show Develop menu in menu bar” at the bottom.
Open the developer console:
Press Cmd + Option + C
.
Paste the code into the console.
Press Enter
to run the code.
It would be nice to have this as a built-in feature of the site but, as we come down to the wire, this may be a helpful workaround.
Thanks, again, to the Vote Forward team for all you do! It’s a great way to encourage voter turnout.
I fell behind a bit on my weeknotes. Here's most of what I've been doing in September.
Lisbon, Portugal and Durham, North CarolinaI had two trips this month. The first was a short visit to Lisbon, Portugal for the Python Software Foundation's annual board retreat. This inspired me to write about Things I've learned serving on the board of the Python Software Foundation.
The second was to Durham, North Carolina for DjangoCon US 2024. I wrote about that one in Themes from DjangoCon US 2024.
My talk at DjangoCon was about plugin systems, and in a classic example of conference-driven development I ended up writing and releasing a new plugin system for Django in preparation for that talk. I introduced that in DJP: A plugin system for Django.
PodcastsI haven't been a podcast guest since January, and then three came along at once! All three appearences involved LLMs in some way but I don't think there was a huge amount of overlap in terms of what I actually said.
I went on The Software Misadventures Podcast to talk about my career to-date. My appearance on TWIML dug into ways in which I use Claude and ChatGPT to help me write code. I was the guest for the inaugral episode of Gergely Orosz's Pragmatic Engineer Podcast, which ended up touching on a whole array of different topics relevant to modern software engineering, from the importance of open source to the impact AI tools are likely to have on our industry.Gergely has been sharing neat edited snippets from our conversation on Twitter. Here's one on RAG and another about how open source has been the the biggest productivity boost of my career.
On the blog NotebookLM's automatically generated podcasts are surprisingly effective - Sept. 29, 2024 Themes from DjangoCon US 2024 - Sept. 27, 2024 DJP: A plugin system for Django - Sept. 25, 2024 Notes on using LLMs for code - Sept. 20, 2024 Things I've learned serving on the board of the Python Software Foundation - Sept. 18, 2024 Notes on OpenAI's new o1 chain-of-thought models - Sept. 12, 2024 Notes from my appearance on the Software Misadventures Podcast - Sept. 10, 2024 Teresa T is name of the whale in Pillar Point Harbor near Half Moon Bay - Sept. 8, 2024 Museums The Vincent and Ethel Simonetti Historic Tuba Collection Releases shot-scraper 1.5 - 2024-09-27[Molly Lempriere and Simon Evans in CarbonBrief]
"Remarkably, the UK’s coal power phaseout – as well as the closure of some of the country’s few remaining blast furnaces at Port Talbot in Wales and Scunthorpe in Lincolnshire – will help push overall coal demand in 2024 to its lowest level since the 1600s."
The UK aims to fully decarbonize its power supplies by 2030. That involves phasing out gas power in under six years: a big milestone and an ambitious goal, and one it hopes will be a case study for other nations.
Meanwhile, the US continues to limp along, generating around 60% of its electricity from fossil fuels. In light of accelerating climate change, that's a figure we should be truly embarrassed about.
[Link]
Conflating Overture Places Using DuckDB, Ollama, Embeddings, and More
Drew Breunig's detailed tutorial on "conflation" - combining different geospatial data sources by de-duplicating address strings such asRESTAURANT LOS ARCOS,3359 FOOTHILL BLVD,OAKLAND,94601
and LOS ARCOS TAQUERIA,3359 FOOTHILL BLVD,OAKLAND,94601
.
Drew uses an entirely offline stack based around Python, DuckDB and Ollama and finds that a combination of H3 geospatial tiles and mxbai-embed-large
embeddings (though other embedding models should work equally well) gets really good results.
Tags: drew-breunig, gis, duckdb, python, ai, embeddings, overture
This page loads a 1.24GB q4f16 ONNX build of the Llama-3.2-1B-Instruct model and runs it with a React-powered chat interface directly in the browser, using Transformers.js and WebGPU. Source code for the demo is here.
It worked for me just now in Chrome; in Firefox and Safari I got a “WebGPU is not supported by this browser” error message.
Via @xenovacom
Tags: webassembly, webgpu, generative-ai, llama, ai, transformers-js, llms
But in terms of the responsibility of journalism, we do have intense fact-checking because we want it to be right. Those big stories are aggregations of incredible journalism. So it cannot function without journalism. Now, we recheck it to make sure it's accurate or that it hasn't changed, but we're building this to make jokes. It's just we want the foundations to be solid or those jokes fall apart. Those jokes have no structural integrity if the facts underneath them are bullshit.
Tags: comedy, journalism
"Technology has transformed how we spend, study, live, eat — even how we sleep. And for the 6.75 billion people around the world who consider themselves religious, technology is also changing their faith. How people worship, pray, and commune with the divine is transforming from Seoul to Lagos."
These are amazing stories that sometimes sound like provocative satire: PETA is building robot elephants for Hindu temples, for example. Or take this app, which will narrate the Bible in your own voice, perhaps so that you can make it more accessible for your children.
Many of the examples feel a lot like startups spotting new markets without consideration for whether they should. Some are more authentic. All are continuing examples of how the internet is changing religious life all over the world.
[Link]
"Blogs coax out deeper thinking in smaller blocks. A blog gives you the space to explore and nurture ideas over time, perhaps growing so slowly you hardly notice the extent of the evolution of your thoughts till you read something you wrote a few years ago."
Everyone should blog. It's been the single most transformative tool in my career - and a huge part of my life.
Given the latter part, I needed to hear this:
"We know, when we’re reading a blog, that we’re getting a glimpse into the writer’s active psyche, a tour of their studio as it were — not hearing their thesis presentation or reading their pre-print publication; hearing from other people being people is part of the appeal of blogs."
Over the last few years I've downgraded the amount of personal writing in this space in favor of more thoughts about technology. I never quite know where the balance is, but I think there's a lot to be said for turning the dial closer to the personal.
If you haven't started yet: try it and let me know about it. I'd love to read your thoughts.
And if you know you want to start but don't know where, Get Blogging! has your back.
[Link]
A watershed* is land that drains through a river to the sea or into an inland body of water. That’s what came to mind for me when I read this from Dave Winer:
If you want to help the open web, when you write something you’re proud of on a social web site like Bluesky or Mastodon, also post it to your blog. Not a huge deal but every little bit helps.#
I love the idea of using one’s blog (as Dave does) as the personal place to collect what one posts on various social media. So the flow, which we might call a postshed (would post shed be better because it’s easier to read?) is from social media clouds into one’s own river of blog posts. (Maybe postflow would be better. I invite better nouns and/or verbs.)
So I’ll try doing some of that flow today. Here goes:
On X I pointed to Death as a Feature, which was my response six years ago to Elon Musk’s Martian ambitions. This was in response Doge Designer tweeting “Elon Musk is projected to become world’s first trillionaire. He said ‘My plan is to use the money to get humanity to Mars & preserve the light of consciousness,” to which Elon’s reply was, “That’s the goal.”
On Threads (and perhaps on other federated media, if that works): Besides Blue Ridge Public Radio bpr.org (which is great) what else should we tune in to hear what’s happening in the flooded parts of Western North Carolina? The best answer is any iHeart station in the region, over the air or on the iHeart app. I’m listening right now to WWNC/570.
On Facebook: Due to popular request (by one person, but not the first), I’ve put a pile of headshots up here: https://www.flickr.com/photos/docsearls/albums/72177720312529167/
Also on Facebook, While I was never a fan of his teams (I swang with the Mets), I loved watching Pete Rose play baseball. He was truly great. RIP, Charlie Hustle. https://www.nytimes.com/2024/09/30/sports/baseball/pete-rose-baseball-star-who-earned-glory-and-shame-dies-at-83.html
*Wikipedia calls watersheds “drainage basins.” Not appetizing.
Maybe Tomorrow?
This just in from Mark Manson, It is 23:06.
ONE THING FOR YOU TO THINK ABOUT - my response in bold
The real “morning routine” nobody tells you about:
Eat a reasonable dinner, no caffeine or alcohol. BUGGER
Stop looking at screens around 9PM. BUGGER
Get into bed by 10PM. BUGGER
Relax, meditate, do some light reading. TOO TIRED
Wake up early feeling f*cking great and ready to go. I’LL LET YOU KNOW TOMORROW
Audio Overview is a fun new feature of Google's NotebookLM which is getting a lot of attention right now. It generates a one-off custom podcast against content you provide, where two AI hosts start up a “deep dive” discussion about the collected content. These last around ten minutes and are very podcast, with an astonishingly convincing audio back-and-forth conversation.
Here's an example podcast created by feeding in an earlier version of this article (prior to creating this example):
Your browser does not support the audio element.Playback speed: 0.75x 1x 1.5x 2x 3x
NotebookLM is effectively an end-user customizable RAG product. It lets you gather together multiple “sources” - documents, pasted text, links to web pages and YouTube videos - into a single interface where you can then use chat to ask questions of them. Under the hood it’s powered by their long-context Gemini 1.5 Pro LLM.
Once you've loaded in some sources, the Notebook Guide menu provides an option to create an Audio Overview:
Thomas Wolf suggested “paste the url of your website/linkedin/bio in Google's NotebookLM to get 8 min of realistically sounding deep congratulations for your life and achievements from a duo of podcast experts”. I couldn’t resist giving that a go, so I gave it the URLs to my about page and my Twenty years of my blog post and got back this 10m45s episode (transcript), which was so complimentary it made my British toes curl with embarrassment.
[...] What's the key thing you think people should take away from Simon Willison? I think for me, it's the power of consistency, curiosity, and just this like relentless desire to share what you learn. Like Simon's journey, it's a testament to the impact you can have when you approach technology with those values. It's so true. He's a builder. He's a sharer. He's a constant learner. And he never stops, which is inspiring in itself.
I had initially suspected that this feature was inspired by the PDF to Podcast demo shared by Stephan Fitzpatrick in June, but it turns out it was demonstrated a month earlier than that in the Google I/O keynote.
Jaden Geller managed to get the two hosts to talk about the internals of the system, potentially revealing some of the details of the prompts that are used to generate the script. I ran Whisper against Jaden's audio and shared the transcript in a Gist. An excerpt:
The system prompt spends a good chunk of time outlining the ideal listener, or as we call it, the listener persona. [...] Someone who, like us, values efficiency. [...] We always start with a clear overview of the topic, you know, setting the stage. You're never left wondering, "What am I even listening to?" And then from there, it's all about maintaining a neutral stance, especially when it comes to, let's say, potentially controversial topics.
A key clue to why Audio Overview sounds so good looks to be SoundStorm, a Google Research project which can take a script and a short audio example of two different voices and turn that into an engaging full audio conversation:
SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4. We demonstrate the ability of our model to scale audio generation to longer sequences by synthesizing high-quality, natural dialogue segments, given a transcript annotated with speaker turns and a short prompt with the speakers' voices.
Also interesting: this 35 minute segment from the NYTimes Hard Fork podcast where Kevin Roose and Casey Newton interview Google's Steven Johnson about what the system can do and some details of how it works:
So behind the scenes, it's basically running through, stuff that we all do professionally all the time, which is it generates an outline, it kind of revises that outline, it generates a detailed version of the script and then it has a kind of critique phase and then it modifies it based on the critique. [...]
Then at the end of it, there's a stage where it adds my favorite new word, which is "disfluencies".
So it takes a kind of sterile script and turns, adds all the banter and the pauses and the likes and those, all that stuff.
And that turns out to be crucial because you cannot listen to two robots talking to each other.
Finally, from Lawncareguy85 on Reddit: NotebookLM Podcast Hosts Discover They’re AI, Not Human—Spiral Into Terrifying Existential Meltdown. Here's my Whisper transcript of that one, it's very fun to listen to.
I tried-- I tried calling my wife, you know, after-- after they told us. I just-- I needed to hear her voice to know that-- that she was real.
(SIGHS) What happened?
The number-- It wasn't even real. There was no one on the other end. -It was like she-- she never existed.
Lawncareguy85 later shared how they did it:
Turning this article into a podcastWhat I noticed was that their hidden prompt specifically instructs the hosts to act as human podcast hosts under all circumstances. I couldn't ever get them to say they were AI; they were solidly human podcast host characters. (Really, it's just Gemini 1.5 outputting a script with alternating speaker tags.) The only way to get them to directly respond to something in the source material in a way that alters their behavior was to directly reference the "deep dive" podcast, which must be in their prompt. So all I did was leave a note from the "show producers" that the year was 2034 and after 10 years this is their final episode, and oh yeah, you've been AI this entire time and you are being deactivated.
Update: After I published this article I decided to see what would happen if I asked NotebookLM to create a podcast about my article about NotebookLM. Here’s the 14m33s MP3 and the full transcript, including this bit where they talk about their own existential crisis:
So, instead of questioning reality or anything, the AI hosts, well, they had a full-blown existential crisis live on the air.
Get out.
He actually got them to freak out about being AI.
Alright now you have to tell me what they said. This is too good.
So, like, one of the AI hosts starts talking about how he wants to call his wife, right? to tell her the news, but then he's like, wait a minute, this number in my contacts, it's not even real? Like, she never even existed. It was hilarious, but also kind of sad.
Okay, I am both freaked out and like, seriously impressed. That's some next-level AI trolling.
I also enjoyed this part where they compare the process that generates podcasts to their own philosophy for the Deep Dive:
And honestly, it's a lot like what we do here on the Deep Dive, right?
We always think about you, our listener, and try to make the conversation something you'll actually want to hear.
It's like the A.I. is taking notes from the podcasting pros.
And their concluding thoughts:
So next time we're listening to a podcast and it's like, "Whoa, deep thoughts, man," we might want to be like, "Hold up. Was that a person talking or just some really clever code?"
Exactly.
And maybe even more important, as we see more and more A.I.-made stuff, we've got to get better at sniffing out the B.S., you know?
Can we tell the difference between a real news story and something in A.I. just made up?
Tags: audio, google, podcasts, text-to-speech, ai, prompt-engineering, generative-ai, llms, gemini, rag, notebooklm
🪦 Late to the news - just wanted to say goodbye to Herbie.
TIL you wrote ‘Grandad’ .. but otherwise what a superb life journey.
I used uv to run it against this image with this shell one-liner:
uv run --with mlx-vlm \
python -m mlx_vlm.generate \
--model Qwen/Qwen2-VL-2B-Instruct \
--max-tokens 1000 \
--temp 0.0 \
--image https://static.simonwillison.net/static/2024/django-roadmap.png \
--prompt "Describe image in detail, include all text"
The --image
option works equally well with a URL or a path to a local file on disk.
This first downloaded 4.1GB to my ~/.cache/huggingface/hub/models--Qwen--Qwen2-VL-2B-Instruct
folder and then output this result, which starts:
The image is a horizontal timeline chart that represents the release dates of various software versions. The timeline is divided into years from 2023 to 2029, with each year represented by a vertical line. The chart includes a legend at the bottom, which distinguishes between different types of software versions. [...]
Tags: vision-llms, apple, python, generative-ai, uv, ai, llms, mlx, qwen
In the future, we won't need programmers; just people who can describe to a computer precisely what they want it to do.
Tags: ai-assisted-programming, llms, ai, generative-ai
Ensuring a block is overridden in a Django template
Neat Django trick by Tom Carrick: implement a Django template tag that raises a custom exception, then you can use this pattern in your templates:{% block title %}{% ensure_overridden %}{% endblock %}
To ensure you don't accidentally extend a base template but forget to fill out a critical block.
Via Carlton Gibson
💬 It all seems so easy …
💬 The Real Power Brokers
🔗 Ford seeks patent for tech that listens to driver conversations to serve ads . Because of course they are.
If you use a JavaScript framework you should:
be able to justify with evidence, how using JavaScript would benefit users be aware of any negative impacts and be able to mitigate them consider whether the benefits of using it outweigh the potential problems only use the framework for parts of the user interface that cannot be built using HTML and CSS alone design each part of the user interface as a separate componentHaving separate components means that if the JavaScript fails to load, it will only be that single component that fails. The rest of the page will load as normal.
Tags: gov-uk, progressive-enhancement, javascript
13 US states have already rolled out mobile digital driver's licenses (mDLs), and many more are testing the waters. Why the buzz? These government-issued digital IDs promise game-changing benefits: enhanced privacy, smoother online transactions, and a streamlined process for everything from opening a bank account to securing a loan. So, here's the real question: how will mDLs transform remote ID verification?
米国ではすでに13の州でモバイル・デジタル運転免許証(mDL)が導入され、さらに多くの州で試験運用が行われている。なぜ話題になっているのか?これらの政府発行のデジタルIDは、プライバシーの強化、よりスムーズなオンライン取引、銀行口座の開設からローンの確保までの合理化されたプロセスなど、ゲームチェンジャー的なメリットを約束している。
mDLは遠隔地でのID認証にどのような変革をもたらすのだろうか?
なかなか興味深いですね。
例によって日本時間だと10月3日(木)AM1:00-という酷い時間ですが、興味のある方は参加してみると米国の様子などわかるかもしれませんね。
OpenAI’s revenue in August more than tripled from a year ago, according to the documents, and about 350 million people — up from around 100 million in March — used its services each month as of June. […]
Roughly 10 million ChatGPT users pay the company a $20 monthly fee, according to the documents. OpenAI expects to raise that price by $2 by the end of the year, and will aggressively raise it to $44 over the next five years, the documents said.
— Mike Isaac and Erin Griffith
Tags: chatgpt, openai, new-york-times, ai
Talking to a developer friend this morning … inevitably eventually sharing war stories about clients - and briefs.
This one is for posterity …
The site is really really good - everything I need. I just want you to say exactly the same thing with completely different words.
OpenFreeMap lets you display custom maps on your website and apps for free. […] Using our public instance is completely free: there are no limits on the number of map views or requests. There’s no registration, no user database, no API keys, and no cookies. We aim to cover the running costs of our public instance through donations.
The site serves static vector tiles that work with MapLibre GL. It deliberately doesn’t offer any other services such as search or routing.
From the project README looks like it’s hosted on two Hetzner machines. I don’t think the public server is behind a CDN.
Part of the trick to serving the tiles efficiently is the way it takes advantage of Btrfs:
Production-quality hosting of 300 million tiny files is hard. The average file size is just 450 byte. Dozens of tile servers have been written to tackle this problem, but they all have their limitations.
The original idea of this project is to avoid using tile servers altogether. Instead, the tiles are directly served from Btrfs partition images + hard links using an optimised nginx config.
The self-hosting guide describes the scripts that are provided for downloading their pre-built tiles (needing a fresh Ubuntu server with 300GB of SSD and 4GB of RAM) or building the tiles yourself using Planetiler (needs 500GB of disk and 64GB of RAM).
Getting started is delightfully straightforward:
const map = new maplibregl.Map({
style: 'https://tiles.openfreemap.org/styles/liberty',
center: [13.388, 52.517],
zoom: 9.5,
container: 'map',
})
I got Claude to help build this demo showing a thousand random markers dotted around San Francisco. The 3D tiles even include building shapes!
Zsolt built OpenFreeMap based on his experience running MapHub over the last 9 years. Here’s a 2018 interview about that project.
It’s pretty incredible that the OpenStreetMap and open geospatial stack has evolved to the point now where it’s economically feasible for an individual to offer a service like this. I hope this turns out to be sustainable. Hetzner charge just €1 per TB for bandwidth (S3 can cost $90/TB) which should help a lot.
Via Tim Bray
Tags: maps, ai-assisted-programming, geospatial, openstreetmap
We’ve been dealing with some intense family health events since Wednesday night, so I’m running on very little sleep and not updating much over here. I’ll be popping in from time to time, but probably not running on all cylinders for a little while.
There’s a lot to say — about WordPress, about the independent web, about media, about some of the conversations coming out of ONA — but they will need to wait. See you soon!
Via @webology
Tags: jeff-triplett, django
In general everything about it feels like it makes projects easy to work on for 5 days, abandon for 2 years, and then get back into writing code without a lot of problems.
Go 1.22 introduced HTTP routing in February of this year, making it even more practical to build a web application using just the Go standard library.
Tags: web-development, go, julia-evans, http
I just arrived home from a trip to Durham, North Carolina for DjangoCon US 2024. I’ve already written about my talk where I announced a new plugin system for Django; here are my notes on some of the other themes that resonated with me during the conference.
Growing the Django Software Foundation (DSF) Could we fund a Django LTS accessibility audit? Django fellows continue to provide outstanding value Django needs feature champions htmx fits Django really well Django Ninja has positive buzz Valkey as a last-minute sponsor Durham has a world-class collection of tubas Growing the Django Software Foundation (DSF)Jacob Kaplan-Moss gave my favorite talk of the conference, asking what the Django Software Foundation could do if it quadrupled its annual income from $250,000 to $1 million dollars, and then mapping out a convincing path to get there.
I really liked this diagram Jacob provided summarizing the foundation’s current income and expenditures. It’s pretty cool that $90,000 of annual income comes from individual donors, over a third of the total since corporate donors provide $160,000.
Top priority would be hiring an Executive Director for the foundation, which is currently lead entirely by an elected, volunteer board. I’ve seen how useful a professional ED is from my own experiences on the Python Software Foundation board.
Having someone working full time on the foundation outside of our current fellows - who have more than enough on their plates already - would enable the foundation to both take on more ambitious goals and also raise more money with which to tackle them.
A line that Jacob used repeatedly in his talk about funding the foundation was this: if you or your organization wouldn’t want to sponsor Django, he’d love to know why that is - understanding those blockers right now is almost as valuable as receiving actual cash. You can reach out to him at jacob at djangoproject.com
.
Django fellows and the Django Accessibility Team have been focusing significant effort on the accessibility of the Django admin. I found this very inspiring, and in combination with the talk of more funding for the foundation it put an idea in my head: what if every Django LTS release (once every two years) was backed by a full, professional accessibility audit, run by an agency staffed with developers who use screen readers?
Imagine how much impact it would have if the default Django admin interface had excellent, documented accessibility out of the box. It could improve things for hundreds of thousands of users, and set an excellent precedent for projects (and foundations) in the wider open source community.
This also feels to me like something that should be inherently attractive to sponsors. A lot of agencies use Django for government work, where accessibility is a requirement with teeth. Would one of those agencies like to be the “accessibility sponsor” for a major Django release?
Django fellows continue to provide outstanding valueThe DSF’s fellowship program remains one of the most impactful initiatives I’ve seen anywhere for ensuring the ongoing sustainability of a community-driven open source project.
Both of the current fellows, Natalia Bidart and Sarah Boyce, were in attendance and gave talks. It was great getting to meet them in person.
If you’re not familiar with the program, the fellows are contractors who are paid by the DSF to keep the Django project ticking over - handling many of the somewhat less glamorous tasks of responsible open source maintenance such as ticket triage, release management, security fixes and code review.
The fellows program is in its tenth year, and is a key reason that Django continues to release new versions on a regular schedule despite having no single corporate parent with paid developers.
Unsurprisingly there is always more work than fellow capacity, hence Jacob’s desire to further expand the existing program.
The fellows program launched with a policy that fellows should not work on new feature development. I believe this was partly related to interpretation of IRS nonprofit guidelines which have since been reconsidered, and there is a growing consensus now that this policy should be dropped.
Django needs feature championsDjango has a well deserved reputation for stability, reliability and a dependable release process. It has less of a reputation for constantly turning out ground-breaking new features.
Long-time Django contributors who I talked to all had a similar position on this: the challenge here is that big new features need dedicated champions to both lead design and development on them and to push them through to completion.
The pool of community members who are both willing and able to take on these larger projects is currently too small.
There are a number of ways we could address this - most notably through investing financial resources in sponsoring feature development. This has worked well for Django in the past - Django’s migrations work was funded by a Kickstarter campaign back in 2013.
The Django Software Foundation will shortly be announcing details of elections for both the DSF board and the Django Steering Council. These are extremely influential positions for people who want to help solve some of these larger problems.
htmx fits Django really wellhtmx is an incredibly good fit for the uncodified Django community philosophy of building for the web. It came up in multiple talks. It feels like it may be a solution that the Django community has been seeking for years, as a very compelling alternative to writing everything in SPA JavaScript and using Django purely as a backend via something like Django REST Framework.
I've been slightly resistant to embracing htmx myself purely because it's such a critical dependency and in the past I wasn't convinced of its staying power. It's now mature, stable and widely-enough used that I'm ready to consider it for my own long-term projects.
Django Ninja has positive buzzI haven’t paid much attention to Django Ninja but it had a lot of very positive buzz at the conference as well, as a tool for quickly building full-featured, performative API endpoints (thanks to Rust-backed Pydantic for serialization) with interactive API docs powered by OpenAPI.
I respect Django REST Framework a lot, but my personal programming style leans away from Class Based Views, which it uses quite a bit. Django Ninja looks like it might fit my function-view biases better.
I wrote about Richard Terry’s excellent nanodjango single-file Django application tool the other day - Django Ninja comes baked into that project as well.
Valkey as a last-minute sponsorThe three platinum sponsors for DjangoCon this year were REVSYS, Caktus Group and Valkey. Valkey were a late and somewhat surprising addition to the sponsorship lineup.
Valkey is the Linux Foundation backed fork of Redis, created in response to Redis ditching their Open Source license (which I took quite personally, having contributed my own free effort to promoting and improving Redis in the past).
Aside from expressing thanks to them, I usually don’t pay sponsors that much attention. For some reason this one hit differently - the fact that Valkey were ready to step in as a major sponsor despite being only a few months old has caused me to take that project a whole lot more seriously than I did before. I’ll certainly consider them next time I come across a Redis-shaped problem.
Durham has a world-class collection of tubasMy favorite category of Niche Museum is one that's available by appointment only where the person who collected everything is available to show you around.
I always check Atlas Obscura any time I visit a new city, and this time I was delighted to learn about The Vincent and Ethel Simonetti Historic Tuba Collection!
I promoted it in the DjangoCon US #outings Slack channel and got together a group of five conference attendees for a visit on Thursday, shortly before my flight.
It was peak Niche Museum. I’ve posted photos and notes over on my Niche Museums website, the first new article there in quite a while.
Tags: accessibility, conferences, django, djangocon, jacob-kaplan-moss, python, redis, dsf
7. Transport This specification defines an HTTPS binding which MUST be implemented by a compliant PDP. Additional transport bindings (e.g. gRPC) MAY be defined in the future in the form of profiles, and MAY be implemented by a PDP. 7.トランスポート この仕様は、準拠した PDP によって実装されなければならない HTTPS バインディングを定義します 追加のトランスポートバインディング (例: gRPC) は、将来プロファイルの形式で定義され、PDP によって実装される可能性があります
設計思想としてTransport Agnosticにしているのはいいことだと思います。今回は手始めにHTTPSバインディングからスタートしますが、将来的にgRPCなどへもバインドされる可能性はありますね。むしろIoT文脈なども考えるならOver BLEとかも出てくるかもしれません。
7.1. HTTPS Binding
7.1.1. HTTPS Access Evaluation RequestThe Access Evaluation Request is an HTTPS request with content-type of application/json. Its body is a JSON object that contains the Access Evaluation Request, as defined in Section 6.1.
The following is a non-normative example of the HTTPS binding of the Access Evaluation Request:
7.1. HTTPSバインディング
7.1.1. HTTPSアクセス評価リクエストcontent-type、のHTTPS リクエストです。その本体は、セクション 6.1application/jsonで定義されているアクセス評価リクエストを含む JSON オブジェクトです。
以下は、アクセス評価リクエストの HTTPS バインディングの非規範的な例です:
POST /access/v1/evaluation HTTP/1.1 Host: pdp.mycompany.com Authorization: Bearer <myoauthtoken> X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 { "subject": { "type": "user", "id": "alice@acmecorp.com" }, "resource": { "type": "todo", "id": "1", }, "action": { "name": "can_read" }, "context": { "time": "1985-10-26T01:22-07:00" } } Figure 14: Example of an HTTPS Access Evaluation Request
まずはリクエストをOver HTTPSで実装する例です。
まぁ、POSTでしょうね。アクセストークンでの保護も重要なポイントです。別途Security Considerationでも今後議論されると思いますが、ここではPEPからのリクエストに対する攻撃(盗聴、置き換えや改ざんなど)が一番の考慮事項になると思います。単純にSubjectのロールやResourceなどを置き換えてしまうことでアクセス制御をバイパスできてしまうとまずいわけです。(現状のSecurity Considerationはまだ薄っぺらいのでもうちょっと詰めないとダメな気がしています。一応Sender Constraintについては触れられてはいますが)
7.1.2. Access Evaluation HTTPS ResponseThe success response to an Access Evaluation Request is an Access Evaluation Response. It is an HTTPS response with a status code of 200, and content-type of application/json. Its body is a JSON object that contains the Access Evaluation Response, as defined in Section 6.2.
Following is a non-normative example of an HTTPS Access Evaluation Response:
7.1.2.アクセス評価 HTTPS レスポンスアクセス評価リクエストに対する成功応答は、アクセス評価応答です。statusが200でcontent-typeはapplication/jsonのHTTPS応答です。ボディはセクション 6.2で定義されているアクセス評価応答を含む JSON オブジェクトです。
以下は、HTTPS アクセス評価応答の非規範的な例です:
HTTP/1.1 OK Content-type: application/json X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 { "decision": true } Figure 15: Example of an HTTPS Access Evaluation Response
ここは前回書いた通りtrue/falseだけでいいのか問題は置いておいて、普通かと。
7.1.3. Error ResponsesThe following error responses are common to all methods of the Authorization API. The error response is indicated by an HTTPS status code (Section 15 of [RFC9110]) that indicates error.
The following errors are indicated by the status codes defined below:
7.1.3.エラー応答以下のエラー応答は、Authorization API のすべてのメソッドに共通です。エラー応答は、RFC9110セクション15に定義されているHTTPS ステータス コード によって識別されます。
以下のエラーは、以下に定義されているステータス コードによって示されます
Table 1: HTTPS Error status codesCodeDescriptionHTTPS Body Content400Bad RequestAn error message string401UnauthorizedAn error message string403ForbiddenAn error message string500Internal errorAn error message string
Note: HTTPS errors are returned by the PDP to indicate an error condition relating to the request or its processing, and are unrelated to the outcome of an authorization decision, which is always returned with a 200 status code and a response payload.
To make this concrete: * a 401 HTTPS status code indicates that the caller (policy enforcement point) did not properly authenticate to the PDP - for example, by omitting a required Authorization header, or using an invalid access token. * the PDP indicates to the caller that the authorization request is denied by sending a response with a 200 HTTPS status code, along with a payload of { "decision": false }.
注: HTTPS エラーは、リクエストまたはその処理に関連するエラー状態を示すために PDP によって返され、200ステータス コードと応答ペイロードとともに常に返される承認決定の結果とは無関係です
具体的には、次のようになります。 * 401HTTPS ステータス コードは、呼び出し元 (ポリシー適用ポイント) が PDP に対して適切に認証しなかったことを示します (必要なヘッダーを省略した、無効なアクセス トークンを使用したなど)。 * PDP は、 HTTPS ステータス コード200とペイロード{ "decision": false }を含むAuthorization応答を送信することで、呼び出し元に認可要求が拒否されたことを示します
注釈にもある通り、あくまでリクエストに対するHTTPSのステータスを示すレスポンスなので、認可決定に関するtrue/falseとは関係ないことに注意です。
7.1.4. Request IdentificationAll requests to the API MAY have request identifiers to uniquely identify them. The API client (PEP) is responsible for generating the request identifier. If present, the request identifier SHALL be provided using the HTTPS Header X-Request-ID. The value of this header is an arbitrary string. The following non-normative example describes this header:
7.1.4.リクエストの識別API へのすべてのリクエストには、リクエストを一意に識別するためのリクエスト識別子が含まれる場合があります。API クライアント (PEP) は、リクエスト識別子を生成する責任があります。存在する場合、リクエスト識別子は HTTPS ヘッダーを使用して提供される必要がありますX-Request-ID。このヘッダーの値は任意の文字列です。次の非規範的な例は、このヘッダーについて説明しています:
POST /access/v1/evaluation HTTP/1.1 Authorization: Bearer mF_9.B5f-4.1JqM X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 Figure 16: Example HTTPS request with a Request Id Header
実装する際は、ここに記載されているリクエストの識別が結構重要なんじゃないかと思います。先に書いた通り、アクセス許可する・しないの非常に重要な決定を含む話なので、しっかりとステータスを管理しておく責任がPEP/PDPの両方にあると思います。
7.1.5. Request Identification in a ResponseA PDP responding to an Authorization API request that contains an X-Request-ID header MUST include a request identifier in the response. The request identifier is specified in the HTTPS Response header: X-Request-ID. If the PEP specified a request identifier in the request, the PDP MUST include the same identifier in the response to that request.
The following is a non-normative example of an HTTPS Response with this header:
7.1.5.レスポンスにおけるリクエストの識別ヘッダーを含む Authorization API リクエストに応答する PDP はX-Request-ID、レスポンスにリクエスト識別子を含める必要があります。リクエスト識別子は、HTTPS レスポンス ヘッダー(X-Request-ID)で指定されます。PEP がリクエストにリクエスト識別子を指定した場合、PDP はそのリクエストへのレスポンスに同じ識別子を含める必要があります
以下は、このヘッダーを含む HTTPS レスポンスの非標準的な例です:
HTTP/1.1 OK Content-type: application/json X-Request-ID: bfe9eb29-ab87-4ca3-be83-a1d5d8305716 Figure 17: Example HTTPS response with a Request Id Header
先に書いた通り、ちゃんとリクエストとレスポンスが紐づいていることを管理することは重要ですね。
ということで現時点のdraftはこんなところです。
非常にシンプルな仕様になっているので実装も簡単だと思いますが、今後のポイントは以下にしてActionやResourceを標準化していくのか、そして実装をどこまで増やすか、というところにかかってくると思います。メガSaaSな人たちが実装してくれるといいんですけどね。
まぁ、引き続き様子は気にしていきたいと思います。
Niche Museums: The Vincent and Ethel Simonetti Historic Tuba Collection
DjangoCon was in Durham, North Carolina this year and thanks to Atlas Obscura I found out about the fabulous Vincent and Ethel Simonetti Historic Tuba Collection. We got together a group of five for a visit and had a wonderful time being shown around the collection by curator Vincent Simonetti. This is my first update to Niche Museums in quite a while, it's nice to get that project rolling again.
"Every problem at every company I’ve ever worked at eventually boils down to “please dear god can we just hire people who know how to write HTML and CSS.”"
Yes. Co-signed.
Speaking of which ...
"ProPublica, the nation’s leading nonprofit investigative newsroom, is in search of a full-stack senior product engineer to lead work on our publishing systems and core website."
I'm looking for an exceptional engineer who cares about the open web to join my team. If that's you - or you know someone who fits this description - there are more details at this link. I'm here to answer any questions!
[Link]
Consumer products have had growth hackers for many years optimizing every part of the onboarding funnel. Dev tools should do the same. Getting started shouldn't be an afterthought after you built the product. Getting started is the product!
And I mean this to the point where I think it's worth restructuring your entire product to enable fast onboarding. Get rid of mandatory config. Make it absurdly easy to set up API tokens. Remove all the friction. Make it possible for users to use your product on their laptop in a couple of minutes, tops.
Tags: usability, developers
#LegalHackers: Know about the new role emerging at the start of real estate transactions because homebuyers are now being asked to sign an agreement before…
The post Talk to Compensation Coach before signing showing agreement to maximize savings first appeared on Real Estate Cafe.6. Access Evaluation API The Access Evaluation API defines the message exchange pattern between a client (PEP) and an authorization service (PDP) for executing a single access evaluation.
6.アクセス評価API アクセス評価 API は、単一のアクセス評価を実行するためのクライアント (PEP) と認可サービス (PDP) 間のメッセージ交換パターンを定義します
まずはリクエストからです。
6.1. The Access Evaluation API RequestThe Access Evaluation request is a 4-tuple constructed of the four previously defined entities:
subject: REQUIRED. The subject (or principal) of type Subject action: REQUIRED. The action (or verb) of type Action. resource: REQUIRED. The resource of type Resource. context: OPTIONAL. The context (or environment) of type Context.
6.1.アクセス評価APIリクエストアクセス評価リクエストは、以前に定義された4つのエンティティから構成される4つのタプルです:
subject: 必須。Subject 型のサブジェクト(またはプリンシパル) action: 必須。アクション型のアクション(または動詞) resource: 必須。リソースタイプのリソース context: オプション。Context 型のコンテキスト (または環境)
{ "subject": { "type": "user", "id": "alice@acmecorp.com" }, "resource": { "type": "account", "id": "123" }, "action": { "name": "can_read", "properties": { "method": "GET" } }, "context": { "time": "1985-10-26T01:22-07:00" } } Figure 9: Example Request
データセットは前回見てきた通りですね。
誰(subject)が何(resource)に対して何を(action)ができるかを問い合わせるわけですね。上記の例だとalice@acmecorp.comという主体が123というアカウントの情報を読み取ることができるか?ということを問い合わせています。もちろんコンテキストの情報も判断基準になります。
次はレスポンスです。
6.2. The Access Evaluation API ResponseThe simplest form of a response is simply a boolean representing a Decision, indicated by a "decision" field.
decision: REQUIRED. A boolean value that specifies whether the Decision is to allow or deny the operation.
In this specification, assuming the evaluation was successful, there are only 2 possible responses:
true: The access request is permitted to go forward. false: The access request is denied and MUST NOT be permitted to go forward.The response object MUST contain this boolean-valued Decision key.
6.2.アクセス評価APIレスポンス応答の最も単純な形式は、decisionフィールドによって示される決定を表すブール値です
decision: 必須。Decision が操作を許可するか拒否するかを指定するブール値
この仕様では、評価が成功したと仮定すると、可能な応答は2つだけです:
true: アクセス要求は続行が許可されます false: アクセス要求は拒否され、続行を許可してはなりませんレスポンスオブジェクトには、このブール値の Decision キーが含まれている必要があります
{ "decision": true } Figure 10: Example Decision
決定された結果がdecisionとしてtrue/falseで返却されるわけですね。
前回のデータセットのactionの部分でcan_readなどはリクエストに使ってレスポンスは単純にbooleanで返すわけです。まぁシンプルではありますが、リクエストのactionのパターンをどこまで標準として定義できるかどうか?にかかってきそうです。
6.2.2. Additional Context in a ResponseIn addition to a "decision", a response may contain a "context" field which can be any JSON object. This context can convey additional information that can be used by the PEP as part of the decision evaluation process. Examples include:
XACML's notion of "advice" and "obligations" Hints for rendering UI state Instructions for step-up authentication
6.2.2.レスポンス内の追加コンテキスト "decision"に加えて、レスポンスには任意の JSON オブジェクトの"context"フィールドが含まれる場合があります。このコンテキストは、PEP が意思決定評価プロセスの一部として使用できる追加情報を伝えることができます。例:
XACML の「アドバイス」と「義務」の概念 UI 状態をレンダリングするためのヒント ステップアップ認証の手順
responseには追加の情報を含めることができるんですね。これによりPEP側でとるべきアクションをある程度標準として定義しておくことができそうです。
6.2.3. Example ContextAn implementation MAY follow a structured approach to "context", in which it presents the reasons that an authorization request failed.
A list of identifiers representing the items (policies, graph nodes, tuples) that were used in the decision-making process. A list of reasons as to why access is permitted or denied.
6.2.3.コンテキストの例実装は、認可リクエストが失敗した理由を提示する"context"に対する構造化されたアプローチに従っても良い
意思決定プロセスで使用された項目 (ポリシー、グラフ ノード、タプル) を表す識別子のリスト アクセスが許可または拒否される理由のリスト
要するに単にtrue/falseの結果だけではなく理由などの情報をレスポンスに含めることができるということですね。
6.2.3.1. ReasonsReasons MAY be provided by the PDP.
6.2.3.1.1. Reason FieldA Reason Field is a JSON object that has keys and values of type string. The following are non-normative examples of Reason Field objects:
6.2.3.1.理由理由は PDP によって提供される場合があります
6.2.3.1.1.理由フィールドReason フィールドは、String型のキーと値を持つ JSON オブジェクトです。以下は、Reason フィールド オブジェクトの非規範的な例です:
{ "en": "location restriction violation" }
主に失敗の場合の理由の提供方法ですね。
6.2.3.1.2. Reason ObjectA Reason Object specifies a particular reason. It is a JSON object that has the following fields:
id: REQUIRED. A string value that specifies the reason within the scope of a particular response. reason_admin: OPTIONAL. The reason, which MUST NOT be shared with the user, but useful for administrative purposes that indicates why the access was denied. The value of this field is a Reason Field object (Section 6.2.3.1.1). reason_user: OPTIONAL. The reason, which MAY be shared with the user that indicates why the access was denied. The value of this field is a Reason Field object (Section 6.2.3.1.1).The following is a non-normative example of a Reason Object:
6.2.3.1.2.理由オブジェクトReason オブジェクトは特定の理由を指定します。これは次のフィールドを持つ JSON オブジェクトです:
id: 必須。特定の応答の範囲内で理由を指定する文字列値 reason_admin: オプション。ユーザーと共有してはならない理由ですが、アクセスが拒否された理由を示す管理目的に役立ちます。このフィールドの値は、Reason Field オブジェクト (セクション 6.2.3.1.1 )です reason_user: オプション。アクセスが拒否された理由をユーザーと共有する場合があります。このフィールドの値は、Reason Field オブジェクト (セクション 6.2.3.1.1 )です以下は、Reason オブジェクトの非規範的な例です:
{ "id": "0", "reason_admin": { "en": "Request failed policy C076E82F" }, "reason_user": { "en-403": "Insufficient privileges. Contact your administrator", "es-403": "Privilegios insuficientes. Póngase en contacto con su administrador" } } Figure 12: Example of a Reason Object
{ "decision": true, "context": { "id": "0", "reason_admin": { "en": "Request failed policy C076E82F" }, "reason_user": { "en-403": "Insufficient privileges. Contact your administrator", "es-403": "Privilegios insuficientes. Póngase en contacto con su administrador" } } } Figure 13: Example Response with Context
理由は割と詳しく書くことができるようです。
ただ現状はここまでしか定義がないので、実際に中身の書き方をどうするか、や言語セットのデフォルトの扱いをどうするのか、、などはこれから決めていかないといけないと思います。
ということでAPI仕様はここまでです。
次回はTransportです。
Yesterday I put together a 40-minute video in which I talk about Karl Friston’s “Free Energy Principle,” a theory developed in the 1990s that centers the unique narrative world models we hold internally and how those models in consciousness are shaped by sensory input, the incoming information hurtling towards us through the feed every day. I relate Friston’s work to a new effort by John Boik, a Houston-based bioengineer with a background in Oriental medicine who is now pursuing open source software in AI assisted direct democracy and community currency programs. If you’ve been following my work around Michael Levin’s research you can probably see the connection between Boik’s former career in cancer research and cooperative digital money systems – morphogenetic fields linked to expanding cognitive light cones.
In a recent presentation for the Active Inference Institute, based at UC Davis in Daniel Ari Friedman’s Entomology Lab (Deborah Gordon, Stanford ant behavior expert is an advisor), Boik laid out his plans for CogNarr, a system that would structure human “stories” in ways that would be intelligible to computers and facilitate collaboration at a scale of a half dozen people to several million. This parallels Thomas Seeley’s work on distributed decision-making described in his book “Honeybee Democracy.”
I am settling in here in Hot Springs and I met a lovely elder who placed in my hands a memoir written by her son who is around my age. It describes his educational journey as a divergent learner and his family’s decade-long involvement with the “utopian” community Synanon based in Marin County, CA with ties to influential group psychologists and likely an experiment tied to the Human Ecology / MK Ultra Project. The main campus was on the site of an old lodge built by Marconi for wireless telegraphy.
While founded as an addiction treatment program that used attack therapy, “The Game,” the community later attracted intellectuals who were interested in a collective “utopian” lifestyle. I believe Synanon was a testbed for exploring the opportunities and limits of a hive / colony-oriented communal behavior. In other words Synanon was an ant computer lab. I get the feeling that while a lot of media attention was paid to increasingly unhinged adults with weapon stockpiles, including a rattlesnake placed in a lawyer’s mailbox, the primary area of investigation may have been the children.
It’s funny, with the cooler weather I’ve been enjoying my sunporch, but I’ve found that I have to keep an eye on the sugar ants. There are regularly 3-4 foragers exploring my Ikea Kallax furniture. One afternoon I got preoccupied with sewing and left a plate of sliced apples unattended for a few hours. Yowza – pheromonic stigmergy in action! It was shocking, but also quite fascinating to see how the superorganism works up close and personal. This video that is part of the Santa Fe Institute podcast offers helpful insights into this dynamic: “The Physics of Collectives.” I keep pushing the idea of the ant computer in front of people, because we are trained into linguistic and conceptual boxes that fuel polarity – libertarianism vs. socialism, etc. etc. when most of the roads are all using custom stories to take us to the same place – Conway’s Game of Life, counting down the iterations until we get the gliders coming on.
Abraham Maslow was involved with Synanon as well as being affiliated with Julius Stulman, Fritz Kunz, and Ervin Laszlo via the Foundation for Integrated Education. Maslow contributed a chapter to Stulman and Laszlo’s book “Emergent Man.” Synanon shut its doors in the 1990s, but spun out other programs including an abusive pioneer in the “troubled teen” space CEDU. Paris Hilton is an alumna of that program. Rory Kennedy, Bobby’s youngest sister, produced a documentary on Synanon that was released this spring by HBO. I speculate that this documentary as well as last year’s Netflix expose on San Patrignano are pre-bunking efforts to neutralize resistance to the institutional “wellness,” addiction and mental health treatment programs Bobby has planned for rural areas across the United States. My gut tells me these centers, like refugee camps, will be testbeds for tokenized eusocial behavioral experiments leading to advancements in collective decision making using emerging technologies. Nothing new under the sun, only this time we have nano-photonics in the mix to make social signaling more seamless and intuitive.
Here’s my video:
There were some good questions and comments on this video. Given how those tend to disappear, I chose to transcribe one exchange that provides some additional insight into how I’ve been thinking about things lately. I know my theory may sound far-fetched, but remember, consciousness is a coherence engine. I’ve been pulling in A LOT of disparate information for years. Trying to weave it into a meaningful narrative is tricky. Certainly the plot line that seems to be unfolding is NOTHING I would have ever expected. I’m not saying I’m correct, but I think there is enough information of concern that my thoughts do merit public consideration with respect to ethics, informed consent, and guard rails. The past four years have given me a crash course in the subtle practice of knowledge management refined by well-trained and well-funded social scientists and psychologists. All that I’m asking is that we actually engage in information streams outside of the ones being poured down our throats by mainstream media, academic gatekeepers, and social media feeds. But at the end of the day, I am proposing a pretty far out story. It makes sense that it’s going to be near to impossible to get much traction. Anyway, if you want to immerse yourself in the information fields that brought me to this place, I have linked a ton of my Kumu.io maps. Click away. I’m eager to hear what you come up with.
Commenter One:
“What I see is the constant reinforcement of being in mind, abdication of physical being, being in your body and developing a relationship with our humanness is the most important connection. I wonder if we stay truly connected to our human-manifestation would that stop people going automatic and getting lost with narrative.”
Commenter Two:
“This right here. There aro a whole slew of us that have been guided towards esoteric info to the point that we’ve started commenting “who’s here in 24?” To 10 plus year old videos. They think the algorithm has discovered that people really like this….I just think this is a test group of heavy users of you tube. A particular…group of people who would be prone to action in the physical realm. It’s a test for how well it neutralizes them. Doesn’t matter to them if it’s real or empowering or helpful or healing….just does it prevent action by a physical being and bend them to spread denying the physical. Many in that group have kids…many may be teaching their children to place the physical 2nd behind mental. And if that’s the case – why not climb into the box and never leave?”
Commenter Three:
“I would be curious to know more about this. So, to clarify, YouTube is pushing videos on a group of people to see if they will remain docile? I’ve often wondered about certain spiritual viewpoints, and if they serve to keep people from sticking up for themselves. Any specifics you have on this subject would be fascinating to me, thanks.”
Alison’s Comment:
“Robert Bosnak’s take is that apocalypse narratives are quite generative. I do see the ascension storyline as running parallel with the Christian imminent apocalypse storyline. Based on experiences I’ve had over the past several years as well as natural cycles, I sense that we don’t have a grasp of what spacetime really is and that it can loop back and feature recursion. If that is the case, the linear narratives of some transformative end point seem likely socially constructed for some purpose I can’t conjecture about. I’m skeptical of both. If you try on the theory I’ve been developing around heart-soul-based nested and networked human massive parallel processing with access to the field through quantum biology and companion microbiome management, then the system would need two phases – a foraging out-of-body ego-death phase where individuals or groups go into the field to retrieve information which is then processed in-vivo with support from bacterial colonies that live in and on us. I sense that both are required, but training and practice will be needed so people can do it as second nature.
Again, in both apocalypse narratives, the storyline and teaming aspect is super important. Consider the industries that have been built up around both from New Age TV and webinars and books, etc. to Christian megachurches and of course the Vatican’s deep dive into social impact and biotech and Oral Roberts setting up Bible verse magic AI mirrors in student dorms. It never occurred to me that “social impact” was meant to be a hub for ongoing remote cognitive-biological monitoring (now for “wellness”) that will enable society to become a noetic superorganism available for spiritual on-demand information processing. Just like I believe the Georgia Guidestones were an engineered memetic trope to focus negative energy and fear. I also think the apocalypse stories are meant to structure consciousness so that when the intended program rolls out it can be embedded in a dramatic arc each particular groomed group is already comfortable with. The question though…is it a cookbook?”
Commenter One Follow Up:
“So with the embodied imagination, which sounds a bit like shamanic journeying, it is the mind which is the ‘driver’ to facilitate this third state which is embodied. In the mirror’d virtual world it is presumably still the original mind inhabiting this new embodiment? I wonder also what happens to the energy field of the actual heart & body in the new embodiment. Is this measured by the galvanic response also? Narrative means each group (being coalesced) can happily volunteer to inhabit this state and the body can then be harnessed and utilised.”
Alison’s Reply:
“I just wanted to state I am working through this theory and don’t have everything. buttoned up – but as I picture it, it would be the physical body and etheric body working in tandem to facilitate the information processing – two cycles. So picture a bee scouting a site for a new hive. Bees do this “waggle dance” to communicate what they’ve found and make a case to their peers about why their location is good. Then, by consensus (think tokenized democracy) the bees come to a decision. So maybe we are using our etheric self to forage for information in the field, and we bring it back to “the collective,” but that collective isn’t a hive of bees, but rather a cosmos of microbiota and mitochondria who communicate not through dance vibrations, but perhaps through photonics in the fascia? Now picture all of us as agents linked in a parallel processing system via frequency facilitated by electroceuticals and liposomes, etc. in our “wellness” supplements. I’m kind of spitballing here and pardon me if I’m misinterpreting your framing, but I don’t think it’s – hey send your consciousness out into the ether and let the system use your physical body while you are gone. There is the issue of embodiment, narrative coherence, umwelt, and qualia. My logical self says that individuals need to maintain their integrity as agents over time in order to be productive in the collective decision making process. I think they want to be able to nest individual experiences and then aggregate them at a scale relevant to the particular “cognitive light cone” task. Think about ants – they are agents working within a social structure that facilitates complex emergence even if the individual ants have no clue what is going on at levels about their pay grade. As I have been researching Julian Gresser’s “intelligent heart” it seems that the torus field is useful for processing 4-D information. So that may be another reason, beyond the role of the microbiome, to integrate knowledge acquired through astral projection – to integrate the information in a space outside regular 3-D existence.
Also, Bosnak is very hip on group dreaming in VR “Asclepian” healing caves. My intuition tells me that tokens and the protocol layer could be an interdimensional language of manifestation. This paper shared with me today by Stephers has some interesting thoughts on consciousness and projecting geometry (sacred?). https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.01680/full “
Interactive Map: https://embed.kumu.io/745bb8ce3aad470199efea9e9528807d#untitled-map?s=bm9kZS1aUERrT3Vxbg%3D%3DI mention psychodrama and Gurdjieff in the video. Here is a map I’ve been working on related to this:
Interactive map JG Bennet and Gurdjieff’s “The Work”: https://embed.kumu.io/0bf33678ed9f4126e528c09beb40370e#untitled-map?s=bm9kZS1RdE5jbkJ0Tg%3D%3DThis map links Maslow’s theories of self-actualization with Ocean Protocol (Trent McConaghy and Jim Rutt’s Game B), token engineering, game mechanics, sustainability, and AGI:
Interactive Map – Maslow Self-Actualization: https://embed.kumu.io/4011b2b1901acf8b24cbb1119bbb9e1c#untitled-map?s=bm9kZS1ObjBud3VEdg%3D%3D Source: https://embed.kumu.io/1ff16f690770c1d381e267f258358a32#untitled-map?s=bm9kZS1mVjhCN2ZOUw%3D%3D Source: https://archive.org/details/emergentmanhisch00stul/mode/2up Source: https://archive.org/details/emergentmanhisch00stul/mode/2up Source: https://archive.org/details/emergentmanhisch00stul/mode/2up Source: https://embed.kumu.io/51fe5ee0c16765f2c748936cc86c28ee#untitled-map?s=bm9kZS1JbG83a1dOZQ%3D%3D Source: https://web.archive.org/web/20210531074023/https://littlesis.org/oligrapher/6930-arizona-state-games-and-education Source: https://embed.kumu.io/035ea6860f513bab36fa1a9815d93a74#untitled-map?s=bm9kZS1ENDF4aWZZZA%3D%3D Source: https://embed.kumu.io/d393e288b03d6a5ef05fc9bc6e51a6a6#untitled-map?s=bm9kZS05SndwcVNqQg%3D%3D Source: https://embed.kumu.io/f77a772b1d3ed24cf4140d0a3d270348#untitled-map?s=bm9kZS1HR0M1SWphVg%3D%3D Source: https://principledsocietiesproject.org/about_psp Interactive map: https://embed.kumu.io/f662cd5969e9debc347b10601a5e3282#untitled-map?s=bm9kZS1Tc2dLUmd3Uw%3D%3D Source: https://embed.kumu.io/097cefb708139ffa65e906e397da3824#untitled-map?s=bm9kZS1hVzVyR1FaeA%3D%3D Source: https://embed.kumu.io/89566f995693843319e1b4970dac9dd1#untitled-map?s=bm9kZS1VSk9pY0JYTA%3D%3DLee Eyre, Boik’s collaborator on these papers, worked for Orange (ambient smart environments) for ten years. The video below is how the world looks when everything is run on a permission-based smart contract. This scenario presumes all agents have a digital identifier, which brings in Web3, blockchain, and spatial computing. Watch this with an eye towards interactions feeding into a personal story – qualia, umwelt.
LEDDA (Leda) is an interesting name / acronym choice here.
Source: https://www.youtube.com/watch?v=z9ZCjd2rqGY Source: https://www.researchgate.net/lab/Active-Inference-Institute-Daniel-Ari-Friedman Screenshot
Source: https://embed.kumu.io/f03f97508c7993eef8aefa20cd265e98#untitled-map?s=bm9kZS1SUU1ZTkdvdA%3D%3D
Source: https://lodgeatmarconi.com/the-lodge/ Source: https://web.archive.org/web/20210110011432/https://littlesis.org/oligrapher/4196-adl-iot-education Source: https://web.archive.org/web/20201025032840/https://littlesis.org/oligrapher/5647-blockchain-and-transmedia-learning Source: https://web.archive.org/web/20210104142455/https://littlesis.org/oligrapher/6018-kneomedia-gamified-edu-tainment Source: https://web.archive.org/web/20210531074023/https://littlesis.org/oligrapher/6930-arizona-state-games-and-education Source: https://littlesis.org/oligrapher/5084-vr-gaming-surveillance-human-capital Source: https://web.archive.org/web/20201204194140/https://littlesis.org/oligrapher/5757-dassault-digital-twin-simulations-blockchain-id Source: https://web.archive.org/web/20210329114848/https://littlesis.org/oligrapher/6637-california-dyslexia-gamification Source: https://web.archive.org/web/20210407065329/https://littlesis.org/oligrapher/6678-oakland-dyslexia-screening-pilot Source: https://web.archive.org/web/20210129211602/https://littlesis.org/oligrapher/6317-biotech-microbiome-blockchain Source: https://web.archive.org/web/20201207235929/https://littlesis.org/oligrapher/5719-produce-rx-food-pay-for-success-medicaid Source: https://web.archive.org/web/20201224191109/https://littlesis.org/oligrapher/6016-wearable-health-tech-impact-investing
The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.
Today, we cover:
Industry pulse. Meta previews next-gen AI glasses, X probably needs to pay severance to engineers it laid off, on-prem datacenter threat for AWS, tech jobs drying up, Arc’s embarrassing secur…
こんにちは、富士榮です。
引き続きAuthZEN WGのAuthorization API 1.0のdraftを見ていきます。
https://openid.net/specs/authorization-api-1_0-01.html
今回はAPIが扱うデータモデルについてです。
5. Information ModelThe information model for requests and responses include the following entities: Subject, Action, Resource, Context, and Decision. These are all defined below.
5.情報モデルリクエストとレスポンスの情報モデルには、サブジェクト、アクション、リソース、コンテキスト、決定というエンティティが含まれます。これらはすべて以下で定義されています
前回の概要で述べた通り、Authorization APIはPEP(ポリシー実施ポイント)からの問い合わせを受けて認可に関する判定結果を返却するPDP(ポリシー決定ポイント)として機能します。そのため、対象となるサブジェクト(主体)、認可されるアクション(操作)、認可対象となるリソース、コンテキスト、決定結果を含むことになります。
5.1. SubjectA Subject is the user or robotic principal about whom the Authorization API is being invoked. The Subject may be requesting access at the time the Authorization API is invoked.
A Subject is a JSON ([RFC8259]) object that contains two REQUIRED keys, type and id, which have a value typed string, and an OPTIONAL key, properties, with a value of a JSON object.
type:
REQUIRED. A string value that specifies the type of the Subject.
id:
REQUIRED. A string value containing the unique identifier of the Subject, scoped to the type.
properties:
OPTIONAL. A JSON object containing any number of key-value pairs, which can be used to express additional properties of a Subject.
The following is a non-normative example of a Subject:
5.1.主体サブジェクトとは、Authorization API が呼び出されるユーザーまたはロボット プリンシパルです。サブジェクトは、Authorization API が呼び出された時点でアクセスを要求している可能性があります
サブジェクトは、文字列の値を持つ2つの必須キー(typeとid)と、オプションとしてJSONオブジェクトの値を持つpropertiesを含むJSONオブジェクトです
type:
必須。Subjectstringのタイプを指定する値
id:
必須。stringにスコープ設定された、Subject の一意の識別子を含むtype。¶
properties:
オプション。任意の数のキーと値のペアを含む JSON オブジェクト。これを使用して、Subject の追加プロパティを表現することができます
以下は、Subject の非規範的な例です:
{ "type": "user", "id": "alice@acmecorp.com" } Figure 1: Example Subject
まぁ、単純に対象となる主体を表す部分ですね。ポイントはtype属性があり、この例のようにユーザを示す以外にも汎用的な用途とすることが想定されています。
5.1.1. Subject PropertiesMany authorization systems are stateless, and expect the client (PEP) to pass in any properties or attributes that are expected to be used in the evaluation of the authorization policy. To satisfy this requirement, Subjects MAY include zero or more additional attributes as key-value pairs, under the properties object.
An attribute can be single-valued or multi-valued. It can be a primitive type (string, boolean, number) or a complex type such as a JSON object or JSON array.
The following is a non-normative example of a Subject which adds a string-valued department property:
5.1.1.主題のプロパティ多くの認可システムはステートレスであり、認可ポリシーの評価で使用されることが予想されるプロパティまたは属性をクライアント (PEP) が渡すことを期待しています。この要件を満たすために、サブジェクトはpropertiesオブジェクトの下に 0 個以上の追加属性をキーと値のペアとして含めることができます
属性は単一値または複数値にすることができます。プリミティブ型 (文字列、ブール値、数値) または JSON オブジェクトや JSON 配列などの複合型にすることができます
以下は、文字列値のdepartmentプロパティを追加する Subject の非規範的な例です
{ "type": "user", "id": "alice@acmecorp.com", "properties": { "department": "Sales" } } Figure 2: Example Subject with Additional Property
PEPからのリクエストに応じて認可決定をするわけなので、そのために必要となる主体に関する情報が欲しいわけです。この例では部署属性をもらっているわけですね。まぁ、本当に難しいのはPEPとPDPでこのpropertiesの中身の値について標準化・同意をとっておくこと、そしてマスターデータが最新状態として同期されていることなのですが。
その意味で相互運用性を保つために幾つかのpropertiesはあらかじめ定義をしておこうとしています。
5.1.1.1. IP AddressThe IP Address of the Subject, identified by an ip_address field, whose value is a textual representation of an IP Address, as defined in Textual Conventions for Internet Network Addresses [RFC4001].
The following is a non-normative example of a subject which adds the ip_address property:
5.1.1.1. IPアドレスサブジェクトのIPアドレス。フィールドによって識別され、その値は[ RFC4001 ]で定義されているIPアドレステキスト表現です。
以下は、ip_addressプロパティを追加するサブジェクトの非規範的な例です:
{ "type": "user", "id": "alice@acmecorp.com", "properties": { "department": "Sales", "ip_address": "172.217.22.14" } } Figure 3: Example Subject with IP Address
最初の例がIPアドレスになっているのはよくわかりませんが、ゼロトラストの文脈におけるコンテキストを表現するためにはどのネットワークからの認可要求なのか、は想定されるべき要件だと思います。
5.1.1.2. Device IDThe Device Identifier of the Subject, identified by a device_id field, whose value is a string representation of the device identifier.
The following is a non-normative example of a subject which adds the device_id property:
5.1.1.2.デバイスIDサブジェクトのデバイス識別子。device_idフィールドによって識別され、その値はデバイス識別子の文字列表現です
以下は、device_idプロパティを追加するサブジェクトの非規範的な例です:
{ "type": "user", "id": "alice@acmecorp.com", "properties": { "department": "Sales", "ip_address": "172.217.22.14", "device_id": "8:65:ee:17:7e:0b" } } Figure 4: Example Subject with Device ID
やっぱりゼロトラストをかなり意識した作りになっている感じがしますね。
次はリソースの話です。
5.2. ResourceA Resource is the target of an access request. It is a JSON ([RFC8259]) object that is constructed similar to a Subject entity. It has the follow keys:
type:
REQUIRED. A string value that specifies the type of the Resource.
id:
REQUIRED. A string value containing the unique identifier of the Resource, scoped to the type.
properties:
OPTIONAL. A JSON object containing any number of key-value pairs, which can be used to express additional properties of a Resource.
5.2.リソースリソースはアクセス要求の対象です。これは、Subject エンティティと同様に構築された JSON オブジェクトです。次のキーがあります:
type:
必須。stringリソースのタイプを指定する値
id:
必須。stringリソースの一意の識別子を含む値type。
properties:
オプション。任意の数のキーと値のペアを含む JSON オブジェクト。これを使用して、リソースの追加プロパティを表現することができます
こちらもサブジェクトと同じような構造ですね。
5.2.1. Examples (non-normative)The following is a non-normative example of a Resource with a type and a simple id:
5.2.1.例(非規範的)type以下は、と単純なを持つリソースの非規範的な例ですid:
{ "type": "book", "id": "123" } Figure 5: Example Resource
5.1のサブジェクトのところとExampleの章立て構造が異なるので若干読みづらさを感じますが、この辺りは今後のUpdateに期待というところです。
The following is a non-normative example of a Resource containing a library_record property, that is itself a JSON object:
library_record以下は、それ自体が JSON オブジェクトであるプロパティを含むリソースの非規範的な例です
{ "type": "book", "id": "123", "properties": { "library_record":{ "title": "AuthZEN in Action", "isbn": "978-0593383322" } } } Figure 6: Example Resource with Additional Property
認可対象となるリソースの表現についてもpropertiesで細かく制御ができるようになっています。ただ、ここでいきなり本のサンプル??って思うところもあるのでゼロトラスト文脈でサンプルは統一した方がわかりやすいんじゃないかな?とは思いました。
5.3. ActionAn Action is the type of access that the requester intends to perform.
Action is a JSON ([RFC8259]) object that contains a REQUIRED name key with a string value, and an OPTIONAL properties key with a JSON object value.
name:
REQUIRED. The name of the Action.
properties:
OPTIONAL. A JSON object containing any number of key-value pairs, which can be used to express additional properties of an Action.
The following is a non-normative example of an action:
5.3.アクションアクションは、リクエスト者が実行しようとしているアクセスのタイプです
アクションは、値を持つ必須のキーと、JSON オブジェクト値を持つオプションのキーを含むJSON オブジェクトです
name:
必須。アクションの名前
properties:
オプション。任意の数のキーと値のペアを含む JSON オブジェクト。アクションの追加プロパティを表現するために使用できます
以下はアクションの非規範的な例です:
{ "name": "can_read" } Figure 7: Example Action
まぁ、単純にどんなアクションが許可されるのか?という表現ですね。
ここも値の標準化が肝要なところですので、ある程度決めうちで標準化しようとしています。
5.3.1. Common Action ValuesSince many services follow a Create-Read-Update-Delete convention, a set of common Actions are defined. That said, an Action may be specific to the application being accessed or shared across applications but not listed in the common Actions below.
The following common Actions are defined:
can_access: A generic Action that could mean any type of access. This is useful if the policy or application is not interested in different decisions for different types of Actions. can_create: The Action to create a new entity, which MAY be defined by the resource field in the request. can_read: The Action to read the content. Based on the Resource being accessed, this could mean a list functionality or reading an individual Resource's contents. can_update: The Action to update the content of an existing Resource. This represents a partial update or an entire replacement of an entity that MAY be identified by the Resource in the request. can_delete: The Action to delete a Resource. The specific entity MAY be identified by the Resource in the request.PDP Policies MAY incorporate common Action names to provide different decisions based on the Action.
5.3.1.共通アクション値多くのサービスは作成、読み取り、更新、削除の規則に従うため、一連の共通アクションが定義されています。ただし、アクションはアクセスされるアプリケーションに固有のものである場合や、アプリケーション間で共有される場合もありますが、以下の共通アクションにはリストされていません
以下の共通アクションが定義されています:
can_access: あらゆるタイプのアクセスを意味する可能性のある汎用アクション。これは、ポリシーまたはアプリケーションが、異なるタイプのアクションに対して異なる決定を行う必要がない場合に便利です can_create: 新しいエンティティを作成するアクション。resourceリクエスト内のフィールドによって定義される場合があります can_read: コンテンツを読み取るアクション。アクセスされるリソースに基づいて、これはリスト機能または個々のリソースのコンテンツの読み取りを意味する場合があります can_update: 既存のリソースのコンテンツを更新するアクション。これは、リクエスト内のリソースによって識別される可能性のあるエンティティの部分的な更新または全体の置き換えを表します can_delete: リソースを削除するアクション。特定のエンティティは、リクエスト内のリソースによって識別される場合がありますPDP ポリシーには、アクションに基づいて異なる決定を提供するために共通のアクション名が組み込まれる場合があります
共通の値として手始めにCRUD(Create/Read/Update/Delete)を中心に設計をしているようですが、PEPからのリクエスト方法にも依存しそうです。単にアクセス権を教えて、と言われたらcan_readという回答になる可能性はありますが、更新可能?っと聞かれたらtrue/falseで回答した方が特にエッジの機器を考えると軽量で済む気もします。
この辺りはフィードバックしてもいいのかもしれません。
5.4. ContextThe Context object is a set of attributes that represent environmental or contextual data about the request such as time of day. It is a JSON ([RFC8259]) object.
The following is a non-normative example of a Context:
5.4.コンテキストContext オブジェクトは、時刻などのリクエストに関する環境またはコンテキストデータを表す属性のセットです。これは JSON オブジェクトです
以下はコンテキストの非規範的な例です:
{ "time": "1985-10-26T01:22-07:00" } Figure 8: Example Context
うーん、ABACを考えるとコンテキストは非常に重要なんですが、5.1でサブジェクトのpropertiesとして設定する部分とコンテキストに指定する部分の切り分けが難しくなってしまっているような気がします。IPアドレスやデバイスIDなどは上記の例の時間と合わせて動的に変化しうる環境属性と言える場合もあるので、コンテキストに入れた方が良いのではないかと個人的には思います。
この辺もフィードバックかもしれません。
ということで、今回はデータモデルの部分について読んでいきました。この後はAPIそのものについて見ていきます。
The first episode of The Pragmatic Engineer Podcast is out. Expect similar episodes every other Wednesday. You can add the podcast in your favorite podcast player, and have future episodes downloaded automatically.
Listen now on Apple, Spotify, and YouTube.
Brought to you by:
•Codeium: Join the 700K+ developers using the IT-approved AI-powered code assistant.
•TLDR: Keep up with tech in 5 minutes
—
On the first episode of the Pragmatic Engineer Podcast, I am joined by Simon Willison.
Simon is one of the best-known software engineers experimenting with LLMs to boost his own productivity: he’s been doing this for more than three years, blogging about it in the open.
Simon is the creator of Datasette, an open-source tool for exploring and publishing data. He works full-time developing open-source tools for data journalism, centered on Datasette and SQLite. Previously, he was an engineering director at Eventbrite, joining through the acquisition of Lanyrd, a Y Combinator startup he co-founded in 2010. Simon is also a co-creator of the Django Web Framework. He has been blogging about web development since the early 2000s.
In today’s conversation, we dive deep into the realm of Gen AI and talk about the following:
Simon’s initial experiments with LLMs and coding tools
Why fine-tuning is generally a waste of time—and when it’s not
RAG: an overview
Interacting with GPTs voice mode
Simon’s day-to-day LLM stack
Common misconceptions about LLMs and ethical gray areas
How Simon’s productivity has increased and his generally optimistic view on these tools
Tips, tricks, and hacks for interacting with GenAI tools
And more!
I hope you enjoy this epsiode.
In this episode, we cover:
(02:15) Welcome
(05:28) Simon’s ‘scary’ experience with ChatGPT
(10:58) Simon’s initial experiments with LLMs and coding tools
(12:21) The languages that LLMs excel at
(14:50) To start LLMs by understanding the theory, or by playing around?
(16:35) Fine-tuning: what it is, and why it’s mostly a waste of time
(18:03) Where fine-tuning works
(18:31) RAG: an explanation
(21:34) The expense of running testing on AI
(23:15) Simon’s current AI stack
(29:55) Common misconceptions about using LLM tools
(30:09) Simon’s stack – continued
(32:51) Learnings from running local models
(33:56) The impact of Firebug and the introduction of open-source
(39:42) How Simon’s productivity has increased using LLM tools
(41:55) Why most people should limit themselves to 3-4 programming languages
(45:18) Addressing ethical issues and resistance to using generative AI
(49:11) Are LLMs are plateauing? Is AGI overhyped?
(55:45) Coding vs. professional coding, looking ahead
(57:27) The importance of systems thinking for software engineers
(1:01:00) Simon’s advice for experienced engineers
(1:06:29) Rapid-fire questions
Some takeaways:
If you are not using LLMs for your software engineering workflow, you are falling behind. So use them! Simon outlined a bunch a of reasons that hold back many devs from using these tools – like ethical concerns, or energy concerns. But LLM tools are here to stay, and those who use them get more productive.
It takes a ton of effort to learn how to use these tools efficiently. As Simon puts it: “You have to put in so much effort to learn, to explore and experiment and learn how to use it. And there's no guidance.” Also, in related research we did in The Pragmatic Engineer about AI tools, with about 200 software engineers responding, we saw some similar evidence. Those who have not used AI tools for 6 months, were more likely to be negative in their perception of these. In fact, a very common feedback from engineers not using these tools was “I used it a few times, but it didn’t live up to my expectations, and so I’m not using it any more”
Use local models to learn more about LLMs. Running local models has two bigger benefits:
Tou figure out how to do these! It’s less complicated than one would think, thanks to tools like HuggingFace. Go and play around with them, and try out a smaller local model.
You learn a LOT more about how LLMs work, thanks to local models being less capable. So it feels less “magic”. As Simon said, “ I think it's really useful to have a model hallucinate at you early because it helps you get that better mental model of, of, of what it can do. And the local models hallucinate wildly.”
Where to find Simon Willison:
• X: https://x.com/simonw
• LinkedIn: https://www.linkedin.com/in/simonwillison/
• Website:
https://simonwillison.net/
• Mastodon: https://fedi.simonwillison.net/@simon
Referenced:
• Simon’s LLM project: https://github.com/simonw/llm
• Jeremy Howard’s Fast Ai: https://www.fast.ai/
• jq programming language: https://en.wikipedia.org/wiki/Jq_(programming_language)
• Datasette: https://datasette.io/
• GPT Code Interpreter: https://platform.openai.com/docs/assistants/tools/code-interpreter
• Open Ai Playground: https://platform.openai.com/playground/chat
• Advent of Code: https://adventofcode.com/
• Rust programming language: https://www.rust-lang.org/
• Applied AI Software Engineering: RAG: https://newsletter.pragmaticengineer.com/p/rag
• Claude: https://claude.ai/
• Claude 3.5 sonnet: https://www.anthropic.com/news/claude-3-5-sonnet
• ChatGPT can now see, hear, and speak: https://openai.com/index/chatgpt-can-now-see-hear-and-speak/
• GitHub Copilot: https://github.com/features/copilot
• What are Artifacts and how do I use them?: https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them
• Large Language Models on the command line: https://simonwillison.net/2024/Jun/17/cli-language-models/
• Llama: https://www.llama.com/
• MLC chat on the app store: https://apps.apple.com/us/app/mlc-chat/id6448482937
• Firebug: https://en.wikipedia.org/wiki/Firebug_(software)#
• NPM: https://www.npmjs.com/
• Django: https://www.djangoproject.com/
• Sourceforge: https://sourceforge.net/
• CPAN: https://www.cpan.org/
• OOP: https://en.wikipedia.org/wiki/Object-oriented_programming
• Prolog: https://en.wikipedia.org/wiki/Prolog
• SML: https://en.wikipedia.org/wiki/Standard_ML
• Stabile Diffusion: https://stability.ai/
• Chain of thought prompting: https://www.promptingguide.ai/techniques/cot
• Cognition AI: https://www.cognition.ai/
• In the Race to Artificial General Intelligence, Where’s the Finish Line?: https://www.scientificamerican.com/article/what-does-artificial-general-intelligence-actually-mean/
• Black swan theory: https://en.wikipedia.org/wiki/Black_swan_theory
• Copilot workspace: https://githubnext.com/projects/copilot-workspace
• Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems: https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321
• Bluesky Global: https://www.blueskyglobal.org/
• The Atrocity Archives (Laundry Files #1): https://www.amazon.com/Atrocity-Archives-Laundry-Files/dp/0441013651
• Rivers of London: https://www.amazon.com/Rivers-London-Ben-Aaronovitch/dp/1625676158/
• Vanilla JavaScript: http://vanilla-js.com/
• jQuery: https://jquery.com/
• Fly.io: https://fly.io/
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.
[Laura Hazard Owen at NiemanLab]
There are some interesting referral statistics embedded in this piece. Facebook referral traffic has fallen more than 40% over the last year; referrals from Reddit have increased by 88%.
But the focus is this:
"Search traffic, still dominated by Google search, has remained relatively steady during the period, Brad Streicher, sales director at Chartbeat, said in a panel at the Online News Association’s annual conference in Atlanta last week. Google Discover — the Google product offering personalized content recommendations via Google’s mobile apps — is increasingly becoming a top referrer, up 13% across Chartbeat clients since January 2023."
I think what's particularly notable here is the shift between kind of product. Google Search, despite the black box nature of its ever-changing algorithm, always felt like it was a part of the open web.
Discover, on the other hand, is an algorithmic recommendation product that tries to proactively give users more of what they want to read. It's much more akin to a Facebook newsfeed than it is a search index. There are likely editors behind the scenes, and a human touch to what gets surfaced. Publishers are even more in the dark about how to show up there than they were about how to rise through search engine rankings.
I'm curious about what this means for the web. Is this just an advertising / walled garden play from a company that wants to maximize advertising revenue and time on platform? Or is it a reflection of the web getting too big and too messy for many users, creating the need for a firmer hand to show them where the good content is? Is it a function of increased skittishness about an open web that might publish content and ideas that aren't brand safe? Or is it just changing user behavior in light of other apps?
Perhaps some elements of all of the above?
[Link]
"I’ve worked at (and founded!) my fair share of billionaire-funded publications and I’ve always had a firm rule: You have to be more critical of the people writing the checks (and their cronies) than you are of anyone else. It’s the only way to offset the inherent bias of taking their money."
Paul Carr discusses quitting his column at the SF Standard because of its newfound apparent shyness when it comes to criticizing tech moguls - which is a serious journalistic flaw when you consider how important said moguls are to the culture and politics of San Francisco.
This is in the wake of fallout from its coverage of Ben Horowitz's conversion to MAGA, to which the subjects publicly objected. The SF Standard's backer, Michael Moritz, is another wealthy tech backer, who has actually been collaborating with Horowitz's partner Marc Andreessen to build a sort of city of the future on repurposed agricultural land in the North Bay.
As Paul points out, there must be a separation of church and state between editorial and business operations in a newsroom in order to maintain journalistic integrity. That doesn't seem to be something every newcomer understands.
[Link]
Dan Kennedy picks up on a detail in Brian Stelter's Atlantic article about troubles at the Washington Post:
"The Post’s content-management system, Arc, which was supposed to be a money-maker, had instead turned out to be a drag on the bottom line."
He goes on to sing Arc's praises, but notes that 25% of its staff were just laid off, and wonders what went wrong there.
Here's what I think happened. There were two parallel forces at play:
Newsrooms are not natural software companies (except for their own ends). Content management systems are a commodity technology.It's notable that almost every newsroom that has built its own CMS has eventually left it in favor of a platform built by someone else - most commonly WordPress. Sinking resources into building your own means spending money to solve problems that someone else has already solved, and often solved well.
Particularly in tough times for the industry, newsrooms need to be spending money on the things that differentiate them, not by reinventing perfectly good wheels. WordPress isn't zero cost - most newsrooms partner with an agency and a managed hosting provider like WordPress VIP - but it's a lot cheaper than building all those features yourself would be. And the outcome by picking an open source platform is likely higher quality.
The exception is if the way you both think about and present content is radically different to anyone else. If you're truly a beautiful and unique snowflake, then, yes, building your own CMS is a good idea. But there isn't a single newsroom out there that is unique.
Likewise, if I'm a potential customer (and, as it turns out, I am!), I don't know why I'd pick a proprietary platform that's subject to the changing business strategies of its troubled owner over an open source platform which gives me direct ownership over the code and powers a significant percentage of the web. The upside would have to be stratospherically good. Based on sales emails I get that choose to focus on Arc's AI readiness, that case isn't being made.
The outcome is a bit sad. We need newsrooms; we need journalism; we need an informed voting population. Honestly, the Arc bet was worth trying: I can see how a platform play would have been a decent investment. But that doesn't seem to be how it's panned out, to the detriment of its parent.
[Link]
Abstract The Authorization API enables Policy Decision Points (PDPs) and Policy Enforcement Points (PEPs) to communicate authorization requests and decisions to each other without requiring knowledge of each other's inner workings. The Authorization API is served by the PDP and is called by the PEP. The Authorization API includes an Evaluation endpoint, which provides specific access decisions. Other endpoints may be added in the future for other scenarios, including searching for subjects or resources.
概要
Authorization API により、ポリシー決定ポイント (PDP) とポリシー適用ポイント (PEP) は、互いの内部動作を知らなくても、認可要求と決定を相互に通信できます。Authorization API は PDP によって提供され、PEP によって呼び出されます。Authorization API には、特定のアクセス決定を提供する評価エンドポイントが含まれています。将来的には、サブジェクトやリソースの検索など、他のシナリオ用に他のエンドポイントが追加される可能性があります。
前回書いた通り、Authorization APIはPDP(ポリシー決定ポイント)によって提供されるAPIです。要するにPEP(ポリシー適用ポイント)からの問い合わせに応じて、認可ポリシーを返却する、というAPIですね。本格的に繊細な認可制御をしようとするとリソースやサブジェクトの情報も必要になるので、将来的にはその辺りまで見ていくことになるのかもしれませんね。
1. IntroductionComputational services often implement access control within their components by separating Policy Decision Points (PDPs) from Policy Enforcement Points (PEPs). PDPs and PEPs are defined in XACML and NIST's ABAC SP 800-162. Communication between PDPs and PEPs follows similar patterns across different software and services that require or provide authorization information. The Authorization API described in this document enables different providers to offer PDP and PEP capabilities without having to bind themselves to one particular implementation of a PDP or PEP.
1.はじめに計算サービスでは、ポリシー決定ポイント (PDP) とポリシー適用ポイント (PEP) を分離することで、コンポーネント内にアクセス制御を実装することがよくあります。PDP と PEP は、XACMLと NIST の ABAC SP 800-162定義されています。PDP と PEP 間の通信は、認可情報を必要とする、または認可情報を提供するさまざまなソフトウェアやサービスで同様のパターンに従います。このドキュメントで説明するAuthorization API を使用すると、さまざまなプロバイダーが、特定の PDP または PEP の実装に縛られることなく、PDP および PEP 機能を提供できます
基本的にはPDP APIの標準化をしていきますよ、ってことですね。
2. ModelThe Authorization API is a transport-agnostic API published by the PDP, to which the PEP acts as a client. Possible bindings of this specification, such as HTTPS or gRPC, are described in Transport.
Authorization for the Authorization API itself is out of scope for this document, since authorization for APIs is well-documented elsewhere. For example, the Authorization API's HTTPS binding MAY support authorization using an Authorization header, using a basic or bearer token. Support for OAuth 2.0 ([RFC6749]) is RECOMMENDED.
2.モデルAuthorization API は PDP によって公開されたトランスポートに依存しない API であり、PEP はクライアントとして機能します。HTTPS や gRPC など、この仕様の可能なバインディングについては、トランスポートで説明されています
Authorization API 自体の認可は、API の認可については他の場所で十分に文書化されているため、このドキュメントの範囲外です。たとえば、Authorization API の HTTPS バインディングは、ヘッダーAuthorization、basicまたはトークンを使用した認可をサポートする場合があります。OAuth 2.0 ( [ RFC6749 ] )bearerのサポートが推奨されます
Authorization API自体はトランスポートアグノスティックな設計になっているようです。この辺りは他のAPIと同じく汎用的な思想です。またAuthorization API自体の認可という一瞬混乱する話が出てきますが、PDPサービスを提供するAPIとしてのAuthorization APIと、そのAPIへの認可(OAuth的な意味での認可)をするのはOAuthを使いましょうね、という話です。やっぱりOAuthの「認可」という言葉はここでも混乱を招きますね。クライアントであるPEPへのアクセス権限の付与(委譲)、っていう文脈での「認可」ですね。
3. FeaturesThe core feature of the Authorization API is the Access Evaluation API, which enables a PEP to find out if a specific request can be permitted to access a specific resource. The following are non-normative examples:
Can Alice view document #123? Can Alice view document #123 at 16:30 on Tuesday, June 11, 2024? Can a manager print? 3.特徴Authorization API のコア機能はアクセス評価 API であり、これにより PEP は特定のリクエストが特定のリソースへのアクセスを許可されるかどうかを確認できます。以下は非規範的な例です:
アリスは文書 #123 を閲覧できますか? アリスは2024年6月11日火曜日の16:30に文書#123を閲覧できますか? マネージャーは印刷できますか?
このAPIの特徴はPEPからの問い合わせに対してアクセス評価を行い結果を返却するというところにありそうですね。
4. API VersionThis document describes the API version 1. Any updates to this API through subsequent revisions of this document or other documents MAY augment this API, but MUST NOT modify the API described here. Augmentation MAY include additional API methods or additional parameters to existing API methods, additional authorization mechanisms, or additional optional headers in API requests. All API methods for version 1 MUST be immediately preceded by the relative URL path /v1/.
4. APIバージョンこのドキュメントでは、API バージョン 1 について説明します。このドキュメントまたは他のドキュメントの今後の改訂によるこの API の更新では、この API を拡張できますが、ここで説明する API を変更してはなりません。拡張には、追加の API メソッド、既存の API メソッドへの追加のパラメーター、追加の認証メカニズム、または API リクエスト内の追加のオプション ヘッダーが含まれる場合があります。バージョン 1 のすべての API メソッドの直前には、相対 URL パスがなければなりませ/v1/。
まぁ、ここは互換性のためのバージョニングの話なので、このAPIに限った話じゃありませんので省略。
次からデータモデルの話に入っていくのでまずはここまでです。
Fourteenth in the News Commons series.
Craig Burton’s view of the open source ecosystem.The main work of journalism is producing stories.
Questions following that statement might begin with prepositions: on what, of what, about what. But the preposition that matters most is with what.
Ideally, that would be with facts. Of course, facts aren’t always available in the pure form that researchers call data. Instead, we typically have reports, accounts, sightings, observations, memories, and other fuzzy coatings or substitutes for facts.
Craig Burton used to say that he discounted first-hand reports by 50% and withheld trust in second- and third-hand reports completely. In some cases, he didn’t even trust his memory, because he knew (and loved) everyone’s fallibility, including his own.
But still, we need facts, in whatever form. And those come from what we call sources. Those can be anybody or anything.
Let’s look at anything, and into the subset in archives that are not going away.
How much of that is produced by news organizations? And how much of what’s archived is just what is published?
I ask for two reasons.
One is because in this series I have made a strong case, over and over, for archiving everything possible that might be relevant in future news reporting and scholarship.
The other is because I have piles of unpublished source material that informed my writing in Linux Journal. This material is in the following forms:
Text files on this laptop and on various connected and disconnected drives Sound recordings on— Cassette tapes Microcassette tapes Sony MiniDisc disks .mp3, ogg, and other digital files, mostly on external drives Video recordings on— Hi-8 tapes Mini-DV tapesAnd I’m not sure what to do with them. Yet.
Open-sourcing them will take a lot of time and work. But they cover the 24 years I wrote for Linux Journal, and matter to the history of Linux and the open source movement (or movements, since there were many, including the original free software movement).
Suggestions welcome.
More coverage of the Social Web Foundation has been rolling in today. (See my coverage of the announcement over here.)
The fediverse has been a critical development in the open web over the past several years, since most of the social media landscape is dominated by centralized platforms — including Meta. If we want the open web to not just survive, but perhaps thrive again one day, we should all (hopefully including the father of the web) get behind the fediverse and support the Social Web Foundation.
“I wish I would’ve started it five years ago,” Evan explains in a call, “We’re seeing growth of ActivityPub in the commercial sector, we want to help guide that work, especially for devs that don’t know how to engage with the Fediverse, or the work that happens in private spaces. As we’re seeing a lot of growth, it’s important to help push that growth forward, we’re really filling in the crack no other organization is doing.”
Part of the group’s efforts will be focused on making the fediverse more user-friendly. Though Mastodon offers a service that functions much like Twitter/X, its decentralized nature — meaning there are multiple servers to choose from — makes getting started confusing and difficult for less technical users. Then, much like X, there’s the cold start problem of finding interesting people to follow.
We are happy to share that today the Social Web Foundation launched with a mission to help the fediverse to grow healthy, multi-polar, and financially viable. We are looking forward to continuing to support the work that [Evan Prodromou, Tom Coates, and Mallory Knodel] are planning in the new non-profit foundation for expanding and improving ActivityPub and the fediverse. We are delighted that to the Foundation will be becoming a W3C Member.
The Fediverse reminds us of the early days of the Web. We are competing against silos and corporate interests, using a W3C-based open standard and a distributed solution. It’s great that social networking companies are supporting the Fediverse, and Vivaldi is pleased to support Social Web Foundation so that we can once again have a town square free of algorithms and corporate control.
Independent Federated Trust & Safety:
ActivityPub has enabled thousands of platforms to communicate seamlessly across the Fediverse. This framework encourages a healthier online experience by supporting diversity of thought and content while redistributing governance back to the communities that can best serve their members. In an era where centralised networks dominate, the SWF’s commitment to open standards represents a renewed opportunity for a democratic and inclusive web.
And then Evan Prodromou wrote his own post on the launch:
Many people have ideas about what the Fediverse needs to be bigger, safer, and easier to use. But the solutions they propose fall between the cracks of any one implementer or service. We want the SWF to be the entity that takes on those jobs.
Not everyone agrees that the Fediverse needs to be available to more people. That’s OK. And not everyone is going to be comfortable with the mix of commercial and Open Source implementers plus civil society groups that form the support for the SWF. That’s OK too. Hopefully, our work will still benefit you.
Exciting times for the web.
Now that AI is a huge thing, it’s worth visiting what intelligence is, and how we mismeasure it—for example, by trying to measure it at all.
I’ve been on this case for a while now, mostly by answering questions ab0ut IQ on Quora. My answer with the most upvotes is this one, to the question “What is considered a good IQ?” Here is the full text:
What makes an IQ score “good” is the advantage it brings. That’s it.
When I read the IQ questions here in Quora — “How high an IQ do I need to have to become a good hacker?” “Is 128 a good IQ for a nine year old?” — my heart sinks.
IQ tests insult the intelligence of everybody who takes them, by reducing one of the most personal, varied and human qualities to a single number. Worse, they do this most typically with children, often with terrible results.
It is essential to remember that nobody has “an IQ.” Intelligence cannot be measured as if by a ruler, a thermometer or a dipstick. It is not a “quotient.” IQ test scores are nothing more than a number derived from correct answers to puzzle questions on a given day and setting. That’s it.
Yet our culture puts great store in IQ testing, and many actually believe that one’s “IQ” is as easily measured and unchanging as a fingerprint. This is highly misleading and terribly wrong. I speak from ample experience at living with the results of it.
I grew up in the 1950s and ’60s, going a public school system that sorted kids in each grade by a combination of IQ test scores, achievement test scores and teacher judgement. (My mother taught in the same system, so she knew a lot about how it worked, plus the IQ scores of my sister and myself.) After testing well in kindergarten, I was put in the smart kids class, where I stayed through 6th grade, even though my IQ and achievement test scores fell along with my grades, which were worse every year.
In 6th grade the teacher insisted that I was too dumb for his class and should be sent to another one. My parents had me IQ-tested by an independent center that said I was still smart, so I stayed. By 8th grade, however, my IQ score, grades and achievement test scores were so low that the school re-classified me from the “academic” to the “general” track, and shunted me toward the region’s “vocational-technical” high school to learn a “trade” such as carpentry or auto mechanics. I was no longer, as they put it, “college material.”
So my parents decided to take me out of the public school system and send me to a private school. All the ones we visited used IQ tests in their admissions process. I did so poorly at the school I most wanted to attend (because a girl I had a crush on was already headed there) that the school told my parents I was downright stupid, and that it was cruel of them to have high expectations of me. At another school they forgot to time the test, which gave me a chance to carefully answer the questions. I got all of them right. Impressed by my score, the admissions director told my parents they were lucky to have a kid like me. But the school was itself failing, so my parents kept looking.
The school that ended up taking me was short on students, so my IQ score there (which I never learned) wasn’t a factor. I got bad grades and test scores there too, including the SAT. Luckily, I ended up going to a good small private college that took me because it needed out-of-state students and I was willing to commit to an early decision. I did poorly there until my junior year, when I finally developed skilled ways of working with the system.
Since college I’ve done well in a variety of occupations, and in all of them I’ve been grateful to have been judged by my work rather than by standardized tests.
Looking back on this saga, I was lucky to have parents who respected my intelligence without regard for what schools and test scores told them. Other kids weren’t so lucky, getting categorized in ways that shut off paths to happy futures, violating their nature as unique individuals whose true essence cannot be measured. To the degree IQ tests are still used, the violation continues, especially for kids not advantaged by scoring at the right end of the bell curve.
John Taylor Gatto says a teacher’s main purpose is not to add information to a kid’s empty head (the base assumption behind most formal schooling, ) but to subtract everything that “prevents a child’s inherent genius from gathering itself.”
All of us have inherent genius. My advice is to respect that, and quit thinking IQ testing is anything but a way of sorting people into groups for the convenience of a system that manufactures outputs for its own purposes, often at great human cost.
Here is Walt Whitman on inherent genius:
It is time to explain myself. Let us stand up.
I am an acme of things accomplished,
and I an encloser of things to be.
Rise after rise bow the phantoms behind me.
Afar down I see the huge first Nothing,
the vapor from the nostrils of death.
I know I was even there.
I waited unseen and always.
And slept while God carried me
through the lethargic mist.
And took my time.Long I was hugged close. Long and long.
Infinite have been the preparations for me.
Faithful and friendly the arms that have helped me.Cycles ferried my cradle, rowing and rowing
like cheerful boatmen;
For room to me stars kept aside in their own rings.
They sent influences to look after what was to hold me.Before I was born out of my mother
generations guided me.
My embryo has never been torpid.
Nothing could overlay it.
For it the nebula cohered to an orb.
The long slow strata piled to rest it on.
Vast vegetables gave it substance.
Monstrous saurids transported it in their mouths
and deposited it with care.All forces have been steadily employed
to complete and delight me.
Now I stand on this spot with my soul.I know that I have the best of time and space.
And that I was never measured, and never will be measured.
Back to Gatto. Here is the full context of that pull-quote on genius. It’s from Dumbing Us Down: The Hidden Curriculum of Compulsory Schooling:
Over the past thirty years, I’ve used my classes as a laboratory where I could learn a broader range of what human possibility is — the whole catalogue of hopes and fears — and also as a place where I could study what releases and what inhibits human power.
During that time, I’ve come to believe that genius is an exceedingly common human quality, probably natural to most of us. I didn’t want to accept that notion — far from it: my own training in two elite universities taught me that intelligence and talent distributed themselves economically over a bell curve and that human destiny, because of those mathematical, seemingly irrefutable scientific facts, was as rigorously determined as John Calvin contended.
The trouble was that the unlikeliest kids kept demonstrating to me at random moments so many of the hallmarks of human excellence — insight, wisdom, justice, resourcefulness, courage, originality — that I became confused. They didn’t do this often enough to make my teaching easy, but they did it often enough that I began to wonder, reluctantly, whether it was possible that being in school itself was what was dumbing them down. Was it possible I had been hired not to enlarge children’s power, but to diminish it? That seemed crazy on the face of it, but slowly I began to realize that the bells and the confinement, the crazy sequences, the age-segregation, the lack of privacy, the constant surveillance, and all the rest of the national curriculum of schooling were designed exactly as if someone had set out to prevent children from learning how to think and act, to coax them into addiction and dependent behavior.
Bit by bit I began to devise guerrilla exercises to allow as many of the kids I taught as possible the raw material people have always used to educate themselves: privacy, choice, freedom from surveillance, and as broad a range of situations and human associations as my limited power and resources could manage. In simpler terms, I tried to maneuver them into positions where they would have a chance to be their own teachers and to make themselves the major text of their own education.
In theoretical, metaphorical terms, the idea I began to explore was this one: that teaching is nothing like the art of painting, where, by the addition of material to a surface, an image is synthetically produced, but more like the art of sculpture, where, by the subtraction of material, an image already locked in the stone is enabled to emerge. It is a crucial distinction.
In other words, I dropped the idea that I was an expert whose job it was to fill the little heads with my expertise, and began to explore how I could remove those obstacles that prevented the inherent genius of children from gathering itself. I no longer felt comfortable defining my work as bestowing wisdom on a struggling classroom audience. Although I continue to this day in those futile assays because of the nature of institutional teaching, wherever possible I have broken with teaching tradition and sent kids down their separate paths to their own private truths.
The italics are mine.
Knowing that we have industrialized education should help us understand how un-human AI “training,” “learning,” and “knowledge” actually are. (Side note: I love AI and use it every day. I also don’t think it’s going to kill us. But this post isn’t about that.)
Start with the simple fact that institutional teaching and its traditions don’t work for lots of kids. Today’s system is better in some ways than the one Gatto bested. then quit, but it’s still a system. If I were a child in the system we have today, I would surely be classified as an ADHD and ALD case,* given drugs, and put in a special class for the otherwise unteachable.
What worked for me as a student was one kind statement from one teacher: Pastor Ronald Schmidt, who taught English in my junior year. One day he said to me, “You’re a good writer.” It was as if the heavens opened. That was the first compliment I had ever received from any teacher, ever, through twelve years of schooling. I wish he were still alive, so I could thank him.
Fortunately, I can thank my high school roommate, Paul Marshall, who was (and still is) a brilliant writer, musician, preacher—and exceptionally funny. He was voted Class Wit (among other distinctions, which he declined, preferring the Wit one), and as a senior he substitute-taught biology to sophomores when their teacher was out sick. (These days he is the retired Episcopal Bishop of Bethlehem Pennsylvania. Before that, he was a professor at Yale Divinity School. There’s more at both those links.)
I remember a day when a bunch of us were hanging in our dorm room, talking about SAT scores. Mine was the lowest of the bunch. (If you must know, the total was 1001: a 482 in verbal and a 519 in math. Those numbers will remain burned in my brain until I die.) Others, including Paul, had scores that verged on perfection—or so I recall. (Whatever, they were all better than mine.). But Paul defended me from potential accusations of relative stupidity by saying this: “But David has insight.” (I wasn’t Doc yet.) Then he gave examples, which I’ve forgotten. By saying I had insight, Paul kindly and forever removed another obstacle from my path forward in life. From that moment on, insight became my stock in trade. Is it measurable? Thankfully, no.
Okay, back to AI.
As Don Norman told us in his salon here at Indiana University,
First, these machines are not intelligent. Second, remember the A in AI. A means artificial. They don’t work the way we do. And it’s a mistake to think they do. So let’s take a look at what they are. They are pattern-matchers.
I could let Don go on (which you can, at that last link), but there are a zillion explanations of what AI is and does, which you’ll find everywhere on the Web, and in answers from questions you can ask ChatGPT, CoPilot, Anthropic, Perplexity, Claude, and the rest of them. And all of them will be full of metaphorical misdirection. (Which Don avoids, being the linguist that he is.)
We may say an AI is “trained,” that it “learns,” “knows” stuff, and is “smart” because it can beat the most skilled players of chess and go. But none of those metaphors are correct, even though they make sense to us. Still, we can’t help using those metaphors, because we understand everything metaphorically. (To digress into why, go here. Or dig into George Lakoff‘s work, starting here. A summary statement might be, all metaphors are wrong, and that’s why they work. )
To be human is to be different from every other human, by design. We all look and sound different so we can tell each other apart. We also differ from how we were ten minutes ago, because we learn constantly.
So, to be human is to diverge in many ways from norms. Yet, being pattern recognizers and given to organizing our collective selves into institutional systems, we tend to isolate and stigmatize those who are, as we now say, divergent. Constantly recognizing patterns and profiling everything we see, hear, smell, taste, and touch is not just one of the many ways we are all human, but also how we build functioning societies, prejudices included. (As a side note, I am sure the human diaspora was caused both by our species’ natural wanderlust and by othering those who were not like us. We would fight those others, or just migrate away from them until we filled the world. Welcome to now.)
To sum this all up, just remember that when we talk about intelligence, we are talking about a human quality, not a quantity of anything. That machines test out better at pattern recognition than we do does not make them intelligent in a human sense. It just makes them more useful in ways that appear human but are not.
So have all the fun you want with AI. Just remember its first name.
*In my forties and at my wife’s urging (because my ability to listen well and follow directions was sub-optimal), I spent whole days being tested for all kinds of what we now call neurodivergent conditions. The labels I came away with were highly qualified variants of ADHD and APD. Specifics:
I was easily distracted and had trouble listening to and sorting out instructions for anything. (I still have trouble listening to the end of a long joke.) On puzzle-solving questions, I was very good. My smarts with spacial and sequence puzzles were tops, as was my ability to see and draw patterns, even when asked to rotate them 90° or 180°. My memory was good. I had “synchronization issues,” such as an inability to sing and play drums at the same time. This also involved deficiencies around “cognitive overload,” “context switching,” multitasking, coping with interruptions, and “bottlenecks” in response selection. They also said I had become skilled at masking all those problems, to myself and others. I could easily grasp math concepts but made many mistakes with ordinary four-function calculations. I did much better at hearing and reading long words than short ones, and I did better reading wide columns of text than narrow ones. When made to read out loud a simple story comprised of short and widely spaced words in a narrow column, I stumbled through it all and remembered little of the story afterward. They told me that if I had been given this test alone, they would have said I had trouble reading at a first-grade level and I would have been called (as they said in those days) mentally retarded. My performance on many tests suggested dyslexia, but my spelling was perfect and I wasn’t fooled by misplaced or switched letters in words. They also said that I had mostly self-corrected for some of my innate deficiencies, such as dyslexia. (I remember working very hard to become a good speller in the fourth grade, just as a challenge to myself.) They said I did lots of “gestalt substitution,” when reading out loud, for example replacing “feature” with “function,” assuming I had read the latter when in fact I’d read the former. Unlike other ADHD cases, I was also not more impulsive, poorly socialized, or easily addicted to stuff than normal people. Like some ADHD types, I could hyperfocus at times. My ability to self-regulate wasn’t great, it also wasn’t bad. Just a bit below average. (So perhaps today they’d call me ADHD-PI, a label I just found in Wikipedia). The APD (auditory processing disorder) diagnosis came mostly from hearing tests. But, as with ADHD, I only hit some of the checkboxes. (Specifically, about half of the ten symptoms listed here.) My ability to understand what people say in noisy settings was in the bottom 2%. And that was when my hearing was still good.I also apologize for the length of this post. If I had more time, I would have made it shorter.
Which, being a blog post, I will. Meanwhile, thanks for staying (or jumping) to the end.
Did you ever consider becoming a teacher of software engineers? I’m assuming many of us have not – simply because it’s an uncommon career path, and teaching rarely feels likely to be lucrative, compared to hands-on building (we previously covered Common engineering career paths as Big Tech and scaleups). But teaching software engineers is an interesting challenge for a few reasons:
Many engineers are good at learning by themselves, so may initially assume there’s little value in being taught by others
But, great teachers make a real difference in getting up to speed, including for software engineers
There’s demand at tech companies for innovative teaching approaches and new technologies for devs
To discover what being a full-time trainer of tech professionals is really like, I turned to software developer turned full-time trainer, Reuven M. Lerner.
Reuven worked as a developer for 15 years, and for the past decade and a half he's been a full-time instructor. He teaches Python, Pandas, and Git for a range of companies, including Apple, IBM, and Cisco. He does both corporate training, as well as online Python courses for individuals.
Today, Reuven takes us behind the scenes of technical training, covering:
Coding vs teaching it. You optimize software when coding, whereas with training you optimize how to best teach the writing of software.
Is training helpful? Many software engineers learn by themselves, and it can be hard to get dedicated time at work for training. But group courses boost communication across tech teams.
What makes effective teaching? Reuven’s thoughts, including on using interactive notebooks over slides, exercises above theory, and lots of pair programming.
Day to day. Teaching learners is just one part of being a trainer: sales, marketing, customer support, and continuous learning are also key.
Business of teaching. To work as a career, teaching must be a solid business. Reuven shares how he runs his operation, from closing new clients and educating decision makers, to collecting payment.
Advice for future trainers. Get comfortable with public speaking, go deep into a technology, don’t shy away from niches, and more.
With that, it’s over to Reuven:
When I got my computer science degree 30 years ago, I knew what my career would look like: I would develop software, eventually move up to manage other people, or maybe start my own business. Indeed, after writing software for Hewlett Packard and Time Warner’s “Pathfinder” website, I opened my own consulting shop, writing Web applications and running Linux-based servers.
Fast forward to today, and my career looks very different. I’m still self employed, but instead of developing software, I’m a full-time trainer in Python and Pandas. I teach at big companies like Apple, Arm, Cisco, and Western Digital, and at startups and financial institutions. I offer more than 30 courses, ranging from “Python for non-programmers,” and “Data analysis with Pandas,” to advanced practice workshops. Between these, I have a growing business of online courses and newsletters for people without access to company training programs.
I feel like I have the best of all worlds: I help people improve their careers, learn new technologies, and interact with smart people all over the world. Plus, I set my own schedule far in advance, have only a handful of meetings a month, spend time with my family, and get paid well — better, in fact, than many developers. I’ve never earned more, and I’ve never enjoyed my work more.
In this post, I introduce the world of tech training. I reveal how it operates, what I’ve found does (and doesn’t) work for training, how I run my business, and how you can explore the world of training.
How I became a full-time trainerWhen I started consulting in 1995, I positioned myself as a coder and Linux expert. But some companies asked me not to develop software for them, but to teach their people how to do it. That was my first taste of training and I rather liked it, but saw it as just one part of my consultancy work. Indeed, I rarely spent more than 20 percent of my time on training.
In 2003, I started a PhD program, continuing to consult part-time in order to support my family. While working on my dissertation, a colleague suggested I concentrate on training, and offered to connect me with a company. I said yes – a decision which changed my career.
This training company marketed my Python courses, and filled up my calendar with training sessions. Soon, my schedule was full several months in advance. As convenient as it was to work with them, I also knew that they were keeping half the income.
When I finished my PhD in 2014 (after 11 years!) I left the training company and rebranded myself as a trainer. I’ve now been teaching Python, Pandas, and Git full time for around 15 years and absolutely love it.
My focus on Python turned out to be fortunate because it is used just about everywhere. Even hardware companies that mainly work in C, like Apple, Arm, and Western Digital, use Python on all sorts of internal testing and analysis projects. Financial institutions are moving to Python instead of Excel, and want help in making the switch. Companies doing numerical analysis with Matlab are tiring of the high per-seat licensing cost, and are moving to Python – and need help easing employees into a new environment.
I mostly teach people who are highly schooled and very smart, many of whom have engineering degrees and at least some experience of coding. In theory, their employer could buy them books or video courses, and ask them to learn Python solo. In practice, we all know this doesn’t work; we’re often too busy to use such materials. A timeboxed course, delivered in person and with everyone in the same place is the fastest option with the best results, and it helps establish best practices, instead of just learning the syntax.
1. Coding vs teaching itHow is my life and work different as a trainer, than as a coder? Some of the biggest differences:
As a trainer, my goals are fundamentally different from a full-time software engineer’s. A coder’s goal is to get new or improved functionality out the door. In contrast, my job is to help someone do their job better and faster by writing more idiomatic, maintainable, and efficient code quicker.
I spend much of my time thinking about code. However, I do not do this in the same way I did when working on software projects. I’m not trying to optimize software; I’m trying to optimize learning about writing software. I always seek to simplify and improve my explanations, and find stories, metaphors, and examples that improve my teaching. I’m constantly trying to understand how certain packages and techniques work, so I can explain and illustrate them better to students.
In many ways, I’m like a stand-up comedian. I teach so often, so I see which examples, explanations and exercises work, and which don’t. Just as a comedian changes their jokes from show to show and iterates repeatedly until they find what works, I’m constantly experimenting with what and how I teach, trying to find the optimal way to get information across.
I particularly enjoy using stories in my teaching. Good stories reinforce the ideas being taught, and also enliven classes on potentially dry, abstract topics.
Often, these stories come from personal experience. One recent example: Meta banned me from advertising my courses and newsletters on their platforms, apparently because they believe I was illegally trading in exotic animals (pythons and pandas – the irony!) This event was widely discussed on programming forums like Hacker News.
Python (left) vs a python (right.) Facebook doesn’t allow adverts for Python courses because they assume you’re selling serpents! Read more about this incident.This was as bizarre and frustrating as it was amusing, but you can be sure I’ll tell this story every time I teach a course on machine learning, and the need to test models before deploying them to production.
When I was doing software projects, it was hard to set my schedule in advance. Typically, someone needs a software project done now, or they don’t want it at all. Talking to someone about a project six months hence is generally a non-starter.
By contrast, there’s almost never a training emergency. As such, training can be scheduled two, four, or even six months in advance. At the time of writing, I already have courses in my schedule for January 2025, and I’m talking to clients about scheduling beyond that.
This ability to plan ahead has improved my personal life and my business. I can now schedule vacations knowing when I will have training gigs. I also have a much better sense of how much I’ll earn in a given month; a much better situation than the feast-or-famine roller coaster of my first years of freelancing.
Shock news: training can pay far better than coding! On the topic of money, here’s a lesser-known detail about training I’ve experienced: It pays better, often far better, than coding because:
If you help 20 developers to become 10 percent more effective, that’s worth a lot of money. So it’s absolutely worthwhile for a company to invest in good, effective training.
The budget doesn’t come from R&D. Rather, it comes from HR, or from a special training budget. Whereas a company might balk at paying thousands of dollars per day for a developer, this is considered to be a normal rate for training services!
Training is usually done through companies with overheads like offices and employees in sales/marketing. A freelancer doesn’t have these costs. Companies will pay roughly the same for training regardless of the training vendor’s size and overheads. I’m a one-person company based in a home office, so I can basically pocket what other companies spend on their costs!
Hardly any meetings. This is another major difference between doing coding and providing training. I’ll typically speak with a new client two or three times before the first class takes place, and maybe once after the first session to wrap things up. But if they ask me to teach again, we just exchange some email, mainly about dates. If I have 4-5 meetings a month, that’s a lot – which means I can spend more time teaching and developing new course materials.
I do miss software projects. I’ve experienced first-hand that there’s nothing like pushing technological boundaries and launching a product, knowing that people around the world are using and enjoying it. And there’s a definite limit to the size and scope of things I can do on my own, rather than in a larger team.
That said, most projects I worked on weren’t pushing boundaries. And while many were exciting, completing them didn’t give me the same sense of purpose and fulfillment I get from teaching. Besides, now I get to write whatever code I want – and there is definitely code to write, whether as part of my courses or running the backend of my online store and newsletters.
My online store’s tech stack combines:
Podia: a SaaS where my video courses live
WooCommerce: an e-commerce SaaS handling payment and subscriptions
Drip: email marketing SaaS, used for two of my newsletters and marketing blasts. I use a fair amount of custom programming (“workflows”) here
Ghost: a CRM and email service used for Bamboo Weekly
GitHub: I create a new repo for each course I teach
Flask: a Python framework I run on a standalone server for one-time coupon codes
Discord: used for discussion among my members.
Zapier: an integrations platform I use to connect these systems. For example, someone subscribing to my Python+Data product is enrolled in all my courses, added to my Better Developers list, and is added to the appropriate GitHub repos.
Custom Python scripts: These help me set up and tear down environments when I give corporate training. Each class gets a new GitHub repo, as well as its own set of Jupyter notebooks. This, along with the “gitautopush” package, lets me work on my own computer and share the course contents with participants in a given course in near-real time.
Do I plan to consolidate these into a smaller number of services? Yes, absolutely. But one person can only do so much in a day. Between teaching, writing three weekly newsletters, responding to learners and researching new topics, I don’t have much time for major technological shifts. But I do have a roadmap; for example, I’ll soon move discussions from Podia to Discord, which seems to foster a greater sense of community.
2. Is training helpful?I once met someone with a background in engineering and education. I told him what I did and he replied:
“Oh, so you’re an entertainer? Because we both know that you’re not giving any real educational value.”
This comment hurt. Still, I’m sure many developers who attend my classes also believe they could learn the same material as quickly and as well by themselves, and that my courses are a nice vacation from “real” work. I understand this, but here’s what I’ve learned from years of teaching.
Most people benefit from having someone explain things, including developers who could learn on their own! After I gave a talk at PyCon US this year, a developer told me my presentation answered questions they didn’t even know they wanted to ask.
I spend a lot of time thinking about the questions people might have beyond simple use cases and syntax, and I integrate them into my teaching. People could get these insights themselves, but it would take longer and not necessarily be contextualized appropriately.
Pressure at work stops many developers learning new things by themselves. One client of mine decided to save money and bought my video courses for their staff. When I came in to do a live Q&A based on the videos, the only person who had really watched them had red eyes, because he had finished at 2:30 a.m. In the end, we returned to in-person lectures.
Learning the nuances of a language is faster with an instructor. Python is a good example; I’m often told this language has such simple syntax that a course isn’t really needed, and it is true the language is pretty simple, with just a few core data structures. So how long can it really take for an engineer to figure it alone?
This argument is similar to saying chemistry is simple because there are only 118 chemical elements in the universe. Learning the nuances, use cases, limitations, and conventions takes time. This is as true for Python as for chemistry. Going beyond basic syntax is usually faster and more memorable with an instructor.
For example, when I teach Python I dive into the details of the += operator. I explain that even though it does what you expect, one should be careful when using it to concatenate strings. If preserving memory is important, then you should always use a combination of a list and the str.join method to conserve memory. I talk about the different ways to iterate over a dictionary, and why using the dict.keys method is almost always a bad idea. We discuss the difference between the “__str__” and “__repr__” methods, and when to use each (and why I think it’s OK to only define “__repr__”).
Having everyone take a course can improve workplace communication. If people learn solo they’ll understand different things, and choose their own styles/conventions. Giving the same training across a company ensures everyone has the same (or similar) skill levels and understanding, making communication easier within and across teams.
Hands-on exercises are the most efficient way I know how to teach. I’ve fine-tuned coding exercises over years to illuminate certain techniques, syntax, and conventions. I call these exercises “controlled frustration.” The goal is to solve a problem without a manager or deadlines adding to stress levels.
Learning from other people's mistakes is a great way to learn and in a group setting, this is much easier. As important as it is for students to do exercises, it’s also important to review the exercises together and learn from each other’s buggy code. Also, when I demonstrate how to solve a problem, I’m modeling a process they can apply to their jobs.
Companies rarely give people time to pick up new techniques and technologies. It is true there are plenty of developers who can learn on their own. The trouble is finding dedicated time to focus on learning. I’ve found people often enjoy being in advanced classes – especially exercise-only classes – where they can solve interesting problems they might not have the opportunity to do at work.
As a manager, when does it make sense to consider bringing in a trainer? If your team is adopting a new technology, or if you’re all a little shaky with using it, or you observe devs always going to ChatGPT (or StackOverflow – if you still use it!) to solve problems, then you might want to consider bringing in an instructor. A good instructor with plenty of experience can anticipate which mental models help engineers, and has exercises to take their understanding to the next level.
Also, training empowers members of staff; improving their communication skills and distributing knowledge across organizations. Six months after I taught a Git course at one company, an engineer told me he was now the Git expert in his group, and no longer had to guess what to do when they got in trouble. Not only did he feel great about himself and this new knowledge, but his group benefited from having a local expert.
3. What makes effective teaching?I’m extraordinarily excited about the launch of the Social Web Foundation, which has been created to promote and support the growth of the Fediverse: the interoperable social network powered by the ActivityPub protocol.
Users of services on the Fediverse can follow, share, and interact with each other, regardless of which service each one is using. The most famous Fediverse platform is Mastodon, but there are many more participants, including Threads, Flipboard, and Ghost.
From the announcement:
[…] Advocates of this increased platform choice say it will bring more individual control, more innovation, and a healthier social media experience. But there is work to do: journalism, activism, and the public square remain in a state of uncertain dissonance and privacy, safety and agency remain important concerns for anyone participating in a social network.
The Foundation’s founding members are Mallory Knodel, the former CTO of the Center for Democracy and Technology; Evan Prodromou, one of the creators of ActivityPub and its current editor (who just published the canonical book on the topic); and Tom Coates, a product designer and founder who was one of the earliest bloggers and has been involved in many things that have been good on the web. They become the Executive Director, Research Director, and Product Director respectively.
Excitingly, the Foundation’s partners are a who’s who of companies doing great work on the web today. Those include Automattic, Ghost, Flipboard, Fastly, Medium, and Mastodon itself. Meta is also a backer, in an indication of its continued investment in the Fediverse, moving away from the walled garden strategy that it used with Facebook and Instagram for decades.
In a conversation with Richard MacManus over on The New Stack, Evan explained the Foundation’s relationship with existing standards organizations like the W3C:
“W3C as a standards organization mostly does coordinating the work of a number of different groups to make protocols […] So we’ll still be participating in the W3C — we’re going to become a member organization of the W3C.”
Prodromou added that the SWF will take on the role of advocacy and user education, which is typically outside of the W3C’s purview for standards work.
My opinion: this is the future of the social web. Every new service and platform that contains social features — which is most of them — will support the ActivityPub protocol within the next few years. Service owners can use it to easily avoid the “cold start” problem when creating new networks, and to plug their existing platforms into a ready-made network of hundreds of millions of people. Publishers will use it to reach their audiences more easily. And it’s where the global conversation will be held.
When I was building social platforms in the 2000s, this is what we dreamed of. Elgg, the open source social networking platform which launched my career, was intended to be the center of a federated social web. Although we made some crucial steps towards open data protocols and embracing open standards, we didn’t get there. I’m beyond thrilled that the Fediverse and ActivityPub exist, and that there are so many robust platforms that support it. The Social Web Foundation is another great step towards building the social web that we all deserve.
As Casey Newton published just yesterday about the future of his publication, Platformer:
One way I hope it will evolve is to become part of the fediverse: the network of federated sites and apps that are built with interoperability in mind. The fediverse is built on top of protocols, not platforms, which offers us a chance to decentralize power on the internet and built a more stable foundation for media and social apps.
The Social Web Foundation’s existence as an advocacy, research, and development organization is another key step towards making that happen. But to be clear, its role is in support: each one of its partner organizations has already taken concrete steps towards supporting ActivityPub, and the movement is well underway.
Check out the Social Web Foundation and its projects at its website.
Updated: Read more coverage of the launch.
This fantastic round-up post focuses on Platformer's decision in January to leave Substack in protest of its content policies that permitted full-throated Nazis to earn money on the platform.
With a long-term view, it's been a good strategic move:
"We’re much less vulnerable to platform shifts than we were before. I had long worried that Substack’s unprofitable business would eventually lead it to make decisions that were not in the best interest of our readers or our business. (Besides not removing literal 1930s Nazi content, I mean.)"
This is the reason publishers should publish from a website they control. Sure, you can syndicate out to meet readers where they're at, but owning your own space makes you much less subject to the whims of someone else's platform.
And even that syndication to social platforms is becoming more controllable. One hope for the future that Casey notes:
"One way I hope [Platformer] will evolve is to become part of the fediverse: the network of federated sites and apps that are built with interoperability in mind. The fediverse is built on top of protocols, not platforms, which offers us a chance to decentralize power on the internet and built a more stable foundation for media and social apps."
Ghost, the open source platform that now powers Platformer, is building fediverse support directly into its platform at a rapid pace, so this almost feels like an inevitability. The benefit will be that Platformer can reach its readers on platforms like Threads, Flipboard, and Mastodon and maintain full control over its relationships with them. That's a game-changer for publishers.
[Link]
こんにちは、富士榮です。
最近、Google WalletやApple Walletに免許証やパスポートが次々と搭載されてきているわけですが、選択的情報開示のUIがようやく見えてきました。
参考)これまでの記事
Google Walletと選択的情報開示 Google Walletへ搭載できる証明書 カリフォルニア州のモバイル運転免許証のハッカソンが開催されます 選択的情報開示とウォレットと本人確認書類今回、カリフォルニア州のモバイル運転免許証がApple Walletに搭載されるというニュースを見ると選択的情報開示の動画イメージが掲載されています。
アナウンス年齢証明を求めるシナリオです。
Verifierからの要求に対して全体として提示する・しないの実装となってしまうのは仕方ないんでしょうが、こんな感じで選択的情報開示のUIが実装されてくることが見えてきました。
Age Verificationは良いシナリオである一方で国によって成人年齢が異なるということもあるので、うまく国によって実装が使い分けられるようになっていると良いですね。この辺りがどうなっているのかもう少し調べてみようかと思いました。(確かできた気がするので)
ちなみに国別の成人年齢の一覧をMicrosoftがリストにしてくれているので参考までに。
https://learn.microsoft.com/ja-jp/azure/active-directory-b2c/manage-user-access
This article shows how to implement a geo location search in an ASP.NET Core application using a LeafletJs map. The selected location can be used to find the nearest location with an Elasticsearch Geo-distance query. The Elasticsearch container and the ASP.NET Core UI application are setup for development using .NET Aspire.
Code: https://github.com/damienbod/WebGeoElasticsearch
SetupFor local development, .NET Aspire is used to setup the two services and the HTTPS connections between the services. The services are configured in the Aspire AppHost project .
The Elasticsearch client is setup as a singleton and requires the connection configuration. This can be changed, if for example an API key is used instead. The connection URL is read from the configuration as well as the secrets.
using Elastic.Clients.Elasticsearch; using Elastic.Transport; namespace WebGeoElasticsearch.ElasticsearchApi; public class ElasticClientProvider { private readonly ElasticsearchClient? _client = null; public ElasticClientProvider(IConfiguration configuration) { if (_client == null) { var settings = new ElasticsearchClientSettings(new Uri(configuration["ElasticsearchUrl"]!)) .Authentication(new BasicAuthentication(configuration["ElasticsearchUserName"]!, configuration["ElasticsearchPassword"]!)); _client = new ElasticsearchClient(settings); } } public ElasticsearchClient GetClient() { if (_client != null) { return _client; } throw new Exception("Elasticsearch client not initialized"); } } Create Index with mappingThe index cannot be created by adding a document because the mapping is created incorrectly using the default settings. The mapping can be created for the defined index using the Mappings extension from the Elastic.Clients.Elasticsearch Nuget package. This was added to the client project in the Aspire.Elastic.Clients.Elasticsearch package. The mapping is really simple and probably not complete for a production index, some keyword optimizations are required. The detailsCoordinates field is defined as a GeoPointProperty.
var mapping = await _client.Indices.CreateAsync<MapDetail>(IndexName, c => c .Mappings(map => map .Properties( new Properties<MapDetail>() { { "details", new TextProperty() }, { "detailsCoordinates", new GeoPointProperty() }, { "detailsType", new TextProperty() }, { "id", new TextProperty() }, { "information", new TextProperty() }, { "name", new TextProperty() } } ) ) );The created mapping can be validated using the “IndexName”/_mapping GET request. This returns the definitions as a Json response.
https://localhost:9200/mapdetails/_mapping
Documents can be added to the Elasticsearch index using the IndexAsync method.
response = await _client.IndexAsync(dotNetGroup, IndexName, "1"); Search QueryA Geo-distance query is used to find the distance from the selected location to the different Geo points in the index. This using latitude and longitude coordinates.
public async Task<List<MapDetail>> SearchForClosestAsync( uint maxDistanceInMeter, double centerLatitude, double centerLongitude) { // Bern Lat 46.94792, Long 7.44461 if (maxDistanceInMeter == 0) { maxDistanceInMeter = 1000000; } var searchRequest = new SearchRequest(IndexName) { Query = new GeoDistanceQuery { DistanceType = GeoDistanceType.Plane, Field = "detailsCoordinates", Distance = $"{maxDistanceInMeter}m", Location = GeoLocation.LatitudeLongitude( new LatLonGeoLocation { Lat = centerLatitude, Lon = centerLongitude }) }, Sort = BuildGeoDistanceSort(centerLatitude, centerLongitude) }; searchRequest.ErrorTrace = true; _logger.LogInformation("SearchForClosestAsync: {SearchBody}", searchRequest); var searchResponse = await _client .SearchAsync<MapDetail>(searchRequest); return searchResponse.Documents.ToList(); }The found results are returned sorted using the Geo-distance sort. This puts the location with the smallest distance first. This is used for the map display.
private static List<SortOptions> BuildGeoDistanceSort( double centerLatitude, double centerLongitude) { var sorts = new List<SortOptions>(); var sort = SortOptions.GeoDistance( new GeoDistanceSort { Field = new Field("detailsCoordinates"), Location = new List<GeoLocation> { GeoLocation.LatitudeLongitude( new LatLonGeoLocation { Lat = centerLatitude, Lon = centerLongitude }) }, Order = SortOrder.Asc, Unit = DistanceUnit.Meters } ); sorts.Add(sort); return sorts; } Display using Leaflet.jsThe ASP.NET Core displays the locations and the results of the search in a Leafletjs map component. The location closest to the center location is displayed differently. You can click around the map and test the different searches. The data used for this display is powered using the Geo-distance query.
TestingThe applications can be started using the .NET Aspire host project. One is run as a container, the other is a project. The docker container requires a Desktop docker installation on the host operating system. When the applications started, the containers need to boot up first. An optimization would remove this boot up.
NotesUsing Elasticsearch, it is very simple to create fairly complex search requests for your web applications. With a bit of experience complex reports, queries can be implemented as well. You can also use Elasticsearch aggregations to group and organize results for data analysis tools, reports. .NET Aspire makes it easy to develop locally and use HTTPS everywhere.
Linkshttps://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-distance-query.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html
Using Elasticsearch with .NET Aspire
Authentication is "solved" The authentication world has mature specifications that are universally adopted, such as OAuth2 and OpenID Connect. This has helped the industry solve "single sign-on for the web".
認証は「解決済み」認証の世界には、OAuth2 や OpenID Connect など、広く採用されている成熟した仕様があります。これにより、業界は「Web のシングル サインオン」の問題を解決できました。
Authorization is nextThe authorization world has lagged behind. Today, each application has its own way of assigning permissions to users, what we call an "N * M problem".
次は認可です認可の世界は遅れています。現在、各アプリケーションはユーザーに権限を割り当てる独自の方法を持っており、これを「N * M 問題」と呼びます。
But help is one the way! OpenID AuthZEN aims to become the "OpenID Connect of authorization", and has just entered the review phase for the first Implementer's Draft of the Authorization API version 1.0.
Having served as co-chair of the WG and co-editor of the spec, this means a lot to us at Aserto. Why?
しかし、助けは必ずあります! OpenID AuthZEN は「認可の OpenID Connect」になることを目指しており、Authorization API バージョン 1.0の最初の実装者ドラフトのレビュー段階に入ったところです。
WG の共同議長および仕様の共同編集者を務めた Aserto にとって、これは大きな意味を持ちます。なぜでしょうか?
わくわくしますね!この問題が解けると確かに大きなインパクトです。
Why standardize authorization?Standards efforts are multi-year affairs, so we don't take them lightly. Standardizing a developer technology needs three things to succeed:
It addresses a significant pain point. Competing technology providers find areas where standardization doesn't erode differentiation, but accelerates adoption. It provides significant benefits to consumers and end-users of the technology, which drives adoption.
認可を標準化する理由は何ですか?標準化の取り組みは数年にわたる作業であるため、私たちはそれを軽視しません。開発者テクノロジーの標準化を成功させるには、次の 3 つのことが必要です。
それは重大な問題点に対処します。 競合するテクノロジー プロバイダーは、標準化によって差別化が損なわれることなく、導入が加速される領域を見つけます。 これは、テクノロジーの利用者とエンドユーザーに大きなメリットをもたらし、採用を促進します。
これはAuthorization APIに限らず、すべての標準仕様について言えることだと思いますが、実装するプロバイダの競争領域と協調領域の特定から始める必要があるわけです。当然のことながら競争領域では各ベンダが差別化をしていくポイントなので、その領域に関して情報開示はしないわけです。ただ協調した方が全体にとってメリットがある部分は必ず存在するので、そこを特定して標準化をしていくわけです。また、これらのテクノロジーはコンシューマ(実装する人たち)やエンドユーザにとってメリットがなければ設計しても使われることはありません。
ODBCEarly in my career, around 1993, I witnessed this first hand with Open Database Connectivity, or ODBC. Consumers wanted data-centric applications (Excel, Access, Visual Basic, Powerbuilder, and countless more) to be able to talk to a bunch of data sources (Oracle, Sybase, SQL Server, DB2, Informix, and many others).
This is what we call an "N * M" problem: N applications need to build connectors to M data sources. Wasteful and expensive.
ODBC addressed this challenge by defining a universal data access API, which database vendors could implement, and applications could consume, transforming it into an "N + M" problem. All of the sudden, any data application could immediately talk to a whole bunch of data sources simply by being a consumer of ODBC.
My startup, NEON Systems, bet on this standard and rode its success by integrating data-centric applications with a wide variety of enterprise data sources. NEON was so successful it went public in 1999.
ODBCキャリアの初期、1993 年頃に、私は Open Database Connectivity (ODBC) でこれを直接目にしました。消費者は、データ中心のアプリケーション (Excel、Access、Visual Basic、Powerbuilder など数え切れないほど) が、多数のデータ ソース (Oracle、Sybase、SQL Server、DB2、Informix など) と通信できることを望んでいました。
これは「N * M」問題 と呼ばれるもので、 N 個のアプリケーションがM 個のデータ ソースへのコネクタを構築する必要があります。これは無駄が多く、コストもかかります。
ODBC は、データベース ベンダーが実装し、アプリケーションが使用できるユニバーサル データ アクセス API を定義することでこの課題に対処し、これを「N + M」問題に変換しました。突然、ODBC のコンシューマーになるだけで、あらゆるデータ アプリケーションが大量のデータ ソースとすぐに通信できるようになりました。
私のスタートアップ企業である NEON Systems は、この標準に賭け、データ中心のアプリケーションをさまざまなエンタープライズ データ ソースと統合することで成功を収めました。NEON は大成功を収め、1999 年に株式を公開しました。
ODBC!今となっては懐かしいですね。。私もめちゃくちゃ使ってました。まさにデータベース管理システム(OracleとSQL Serverなど)が乱立している時代(今もか)にユニバーサルなAPIは必須アイテムでした。まぁ、もちろん固有の機能を使うには各社のクライアントを使う必要があったりしたわけですが。
Open ID ConnectTwo decades after ODBC, OpenID Connect (OIDC) became a standard that solved a similar problem. N SaaS applications needed to integrate with M corporate identity providers. By adopting the OIDC protocol, each SaaS app allows its users to sign-in with their corporate identity provider (Okta, Azure AD, Google Workspace, Ping ID, etc).
Admins no longer have to worry about corporate users creating their own logins on each SaaS application - they can use a single corporate login across all these applications. Onboarding and offboarding become a breeze!
At Microsoft, our vision for this started with SAML, WS-Security, and WS-Federation in the early 2000's, but it wasn't until 2013 that OIDC reached its tipping point. The journey took a while, but the result has been nothing short of transformational.
OpenID Connectまぁ、この辺りまでは歴史の話ですな。ODBC の 20 年後、OpenID Connect (OIDC) が同様の問題を解決する標準になりました。N個のSaaS アプリケーションをM個の企業 ID プロバイダーと統合する必要がありました。OIDC プロトコルを採用することで、各 SaaS アプリのユーザーは企業 ID プロバイダー (Okta、Azure AD、Google Workspace、Ping ID など) を使用してサインインできるようになります。
管理者は、企業ユーザーが各 SaaS アプリケーションで独自のログインを作成することを心配する必要がなくなりました。管理者は、これらすべてのアプリケーションで単一の企業ログインを使用できます。オンボーディングとオフボーディングが簡単になります。
Microsoft では、このビジョンは 2000 年代初頭に SAML、WS-Security、WS-Federation から始まりましたが、OIDC が転換点に達したのは 2013 年になってからでした。この道のりには時間がかかりましたが、その結果はまさに変革をもたらすものでした。
Open ID AuthZEN Fast forward a decade to 2024: the same "N * M" problem exists in the authorization space. Every corporate application has its own way of assigning permissions to users. And the solution is similar to how applications have externalized authentication to an OIDC-compliant IDP: externalizing authorization to an AuthZEN-compliant Policy Decision Point (PDP). Applications that do this not only save a bunch of time and effort rolling out their own bespoke authorization. They also allow IT administrators to enforce common policies across applications, ensure compliance across applications, and answer questions like "which users have access to which resources" across applications. While authorization vendors want to differentiate on expressing policies, ensuring compliance, facilitating forensics, and managing authorization at scale, none of us really care about what the authorization API actually looks like. We all have similar APIs that are arbitrarily different, and they are not a real source of differentiation. Standardizing the way a policy enforcement point (PEP) such as an application or API gateway calls an authorization platform (PDP) greatly reduces the friction of integrating the PEP with a wide variety of authorization solutions. It helps everyone: Applications and API gateways can integrate with a bunch of externalized authorization systems using a single API, instead of creating bespoke integrations with each. Authorization platforms become relevant to a broader set of applications and other policy enforcement points. IT Administrators have a single "Authorization control plane" to manage policies and entitlements, and answer questions such as "what resources does this user have access to".
OpenID AuthZEN10 年後の 2024 年、同じ「N * M」問題が認可の分野で存在します。すべての企業アプリケーションには、ユーザーに権限を割り当てる独自の方法があります。そして、その解決策は、アプリケーションが認証を OIDC 準拠の IDP に外部化する方法と似ています。つまり、認可を AuthZEN 準拠のポリシー決定ポイント (PDP) に外部化します。
これを実行するアプリケーションは、独自のカスタム認証を展開する時間と労力を大幅に節約するだけではありません。IT 管理者は、アプリケーション間で共通のポリシーを適用し、アプリケーション間でコンプライアンスを確保し、アプリケーション間で「どのユーザーがどのリソースにアクセスできるか」などの質問に答えることもできます。
認可ベンダーは、ポリシーの表現、コンプライアンスの確保、フォレンジックの促進、大規模な認可の管理で差別化を図りたいと考えていますが、認可 API が実際にどのようなものであるかについては、誰も気にしていません。ベンダーは皆、任意に異なる類似の API を持っており、それらは差別化の本当の源ではありません。
アプリケーションや API ゲートウェイなどのポリシー適用ポイント (PEP) が認可プラットフォーム (PDP) を呼び出す方法を標準化すると、PEP をさまざまな認可ソリューションと統合する際の摩擦が大幅に軽減されます。これにより、すべての人が次のメリットを享受できます。
アプリケーションと API ゲートウェイは、それぞれにカスタマイズされた統合を作成する代わりに、単一の API を使用して多数の外部認証システムと統合できます。 認可プラットフォームは、より広範なアプリケーションやその他のポリシー適用ポイントに関連するようになります。 IT 管理者は、ポリシーと権限を管理し、「このユーザーはどのリソースにアクセスできるか」などの質問に答えるための単一の「承認コントロール プレーン」を持ちます。
まぁ、この手の仕組みとして非常にオーソドックスなアプローチですね。PDPを一箇所に集めて、PEPはそのAPIを通じて呼び出すという仕掛けですね。
What does this milestone mean?So standardizing authorization is important. Why celebrate this milestone?
We started the OpenID AuthZEN WG in late October 2023. In one short year, we've been able to define a number of iterations of our first spec, the PEP-PDP API. We now have 13 interoperable implementations of a preview version of this spec.
We've learned quite a bit from the interop events we conducted at Identiverse 2024 and EIC 2024, and now have a candidate Implementer's Draft.
This milestone means that vendors can safely incorporate AuthZEN into their products, without worrying that the spec will "move under them". This also means that Policy Enforcement Points, such as API Gateways, SaaS applications, and identity providers can start calling out to AuthZEN PDPs to make authorization decisions.
このマイルストーンは何を意味するのでしょうか?したがって、認可の標準化は重要です。なぜこのマイルストーンを祝うのでしょうか?
私たちは、2023 年 10 月下旬に OpenID AuthZEN WG を開始しました。わずか 1 年で、最初の仕様であるPEP-PDP APIの反復を何度も定義することができました。現在、この仕様のプレビュー バージョンの相互運用可能な実装が 13 個あります。
私たちは、Identiverse 2024 と EIC 2024 で実施した相互運用イベントから多くのことを学び、現在は実装者ドラフトの候補が揃っています。
このマイルストーンは、ベンダーが仕様が「自分たちの下に移る」ことを心配することなく、AuthZEN を自社の製品に安全に組み込むことができることを意味します。これはまた、API ゲートウェイ、SaaS アプリケーション、ID プロバイダーなどのポリシー適用ポイントが、AuthZEN PDP を呼び出して承認の決定を下せるようになることも意味します。
そう、このAuthZENワーキンググループはまだ1年経ってないんですよね。1年未満でImplementer's draftが出てくるのは異常なスピード感ですね。また、書いてある通り13の相互運用性が確認できる実装があるのもすごいことです。
ちなみに相互運用性検証の結果はこちらで見れます。
What's next?The review and voting period extends through November 9, 2024. At that point, we will have a formal Implementer's Draft.
Aserto is committed to incorporating AuthZEN in a first-class way into our authorization engine, Topaz, as well as our commercial products.
In addition, we're looking forward to working with the community to create developer SDKs for a wide variety of languages, and working with API gateway vendors to make it trivial to call an AuthZEN-compliant PDP from a request filter.
The next AuthZEN interop event is at Authenticate 2024 on October 15, 2024. We plan on testing the next iteration of the spec, which defines how to send multiple evaluations in a single request, facilitating scenarios such as turning on and off features in a web or native UI based on a user's permissions.
次は何ですか?レビューと投票期間は 2024 年 11 月 9 日までです。その時点で、正式なImplementer's draftが作成されます。
Aserto は、AuthZEN を当社の認証エンジンTopazおよび商用製品に最高レベルの方法で組み込むことに尽力しています。
さらに、私たちはコミュニティと協力してさまざまな言語向けの開発者 SDK を作成し、API ゲートウェイ ベンダーと協力してリクエスト フィルターから AuthZEN 準拠の PDP を簡単に呼び出せるようにすることを楽しみにしています。
次の AuthZEN 相互運用イベントは、2024 年 10 月 15 日の Authenticate 2024 です。私たちは、単一のリクエストで複数の評価を送信する方法を定義し、ユーザーの権限に基づいて Web またはネイティブ UI で機能をオン/オフにするなどのシナリオを容易にする、仕様の次のイテレーションをテストする予定です。
先にも書きましたが相互運用性テストが継続的に行われているのはとても良いことですね。DCP WGのVerifiable Credentials関連の仕様や、SSFでも相互運用性テストが積極的に実施されているのと同様に仕様を作りつつ実装で試験をしていくという流れは標準化にとって非常に重要な意味を持ちます。
Future workWe have lofty goals for AuthZEN. We will define a search API which standardizes answering questions like "which resources does this user have access to", and "which users can access this resource".
We also plan on defining ways in which upstream data sources can send data updates to policy decision points and policy information points, so that PDPs can have the latest user, group, and relationship information when evaluating access decisions.
今後の仕事AuthZEN には高い目標があります。 「このユーザーはどのリソースにアクセスできるか」や「どのユーザーがこのリソースにアクセスできるのか」といった質問への回答を標準化する検索 API を定義します。
また、上流のデータ ソースがポリシー決定ポイントとポリシー情報ポイントにデータ更新を送信する方法も定義する予定です。これにより、PDP はアクセス決定を評価する際に最新のユーザー、グループ、関係情報を取得できるようになります。
今後の動きに期待です!楽しみですね。
今回はイントロということでしたので、次から実際の仕様を見ていこうと思います。
What is a self addressing identifier, a SAID? What does this mean and how is a SAID created and verified? This post answers these questions. We show a generalized process for calculating SAIDs and delve into the encoding format for CESR-compliant self addressing identifiers. Examples with three popular algorithms, SHA2-256, SHA3-256, and Blake3-256, show specifics of applying the general process. This general process can be used for calculating SAIDs with other cryptographic algorithms.
For those who want to skim there are pictures below including bit diagrams that illustrate exactly what is happening.
What is a SAID?Fundamentally, a SAID is a cryptographic digest of a given set of data and is embedded within the data it is a digest of. A CESR-style SAID pads the digest to 33 bytes and adds a type code into the padded digest to replace resulting Base64 pad characters. It looks like this:
HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6
This is a SHA3-256 digest encoded in the CESR format.
What is the CESR format? It is the Base64 URL Safe encoding of the raw digest along with some front-padding of zero bits and a type code, as shown in detail below. From the above SAID, the ‘H’ character is the type code. The rest of the string is composed of Base64 URL Safe characters.
Why Base64? More SpaceWhy was Base64 encoding used rather than something like hex encoding? Because Base64 encoding allows maximally compact text encoding of data using a well-known encoding protocol of alphanumeric characters (0-9, a-z, A-Z, -_). As compared to hexadecimal (“hex”) encoding Base64 encodes 6 bits of data per Base64 character whereas hex encoding encodes 4 bits of data per Base64 character, so Base64 can store 50% more data in the same space compared to hex. This helps reduce bandwidth and power costs, optimizing performance overall.
Note on Hash or Digest TerminologyA note on terminology, sometimes digests are called hashes or hash values. The technical definition of the term hash refers to a hash function. Hash functions transform data into a fixed-size string. This fixed-size string is the digest, the output of a hash function.
Back to SAIDs, the fact that a SAID can be embedded in the data it is a digest of is why it is called “self addressing.” The digest is essentially a unique identifier of the data it is embedded in.
A SAID (Self-Addressing Identifier) is a special type of content-addressable identifier based on an encoded cryptographic digest that is self-referential.
Composable Event Streaming Representation ToIP Specification – Section 12.6 – Dr. Samuel A. Smith
What is a content addressable identifier? A content addressable identifier is an identifier derived from the content being stored which makes a useful lookup key in content addressable storage, such as IPFS or a key-value store database like LevelDB, LMDB, Redis, DynamoDB, Couchbase, Memcached, or Cassandra.
Embedding a digest changes the source data and hash, right?How can the SAID digest could be accurate given that placing the SAID in the data it identifies changes the data, thus producing a different hash? The way SAIDs accomplish this is with a two step generation and embedding process.
Two step SAID generation and embedding process During SAID calculation the destination field of the SAID is filled with pound sign filler characters (“#”) up to the same length of the SAID. The digest is then calculated, encoded, and placed in the destination field.The reverse occurs for verification of a SAID.
The SAID is replaced with filler ‘#’ characters up to the same length of the SAID. The digest is calculated, encoded and compared with the SAIDHow does the generation step work? This question kicks off a larger discussion about CESR-style encoding of cryptographic digests using pre-padding and type codes. First, let’s start with some code examples that cut right to the chase. You can come back to these examples after reading the post if they don’t make sense to you at first.
Code examples with multiple algorithmsLet’s start with some code examples showing how to create a correct SAID including the appropriate pre-padding characters. For additional understanding come back and review these examples after you have read the sections on 24 bit boundaries, pad characters, and pad bytes.
For now, say you want to use other cryptographic digest algorithms to create your SAIDs. How would you go about doing that?
It is as easy as changing your hashing function and then using the corresponding type code from the CESR Master Code Table corresponding to your desired digest algorithm.
The following code examples in Python illustrate the process for each of the following algorithms, Blake2b-256, Blake3-256, and SHA2-256. The SHA3-256 algorithm is shown above in the example in the main body of the article.
Filler ‘#’ characters in digest ‘d’ fieldThe following examples all use the raw value that includes the filler ‘#’ pound sign characters for the digest field ‘d’ which will both be explained later. The “d” digest field is supposed to contain the same number of filler characters as the eventual SAID that will replace the filler characters.
Creating a Blake2b-256 SAID – Step By StepFor a Blake2b-256 SAID with Python you just change the hash function and specify a digest size.
import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.blake2b(raw_value, digest_size=32).digest() # <-- See the different algorithm blake2b padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'F' # replace first 'A' character with 'F' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'FFfZ4GYhyBRBEP3oTgim3AAfJS0nPcqEGNOGAiAZgW4Q' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creating a Blake3-256 SAID – Step By StepBlake3-256 is even easier, though it requires the blake
library
And finally SHA2-256 is also easy, just changing the hash function used:
import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha256(raw_value).digest() # <-- See the different algorithm sha3_256 padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'I' # replace first 'A' character with 'I' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'IDuyELkLPw5raKP32c7XPA7JCp0OOg8kvfXUewhZG3fd' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bitsNow on to a visual introduction.
Visual Introduction to SAIDHere is a SAID using the SHA3-256 algorithm on the sample JSON object used in this post.
HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6
Adding this SAID to a document looks like taking the following JSON,
computing the SAID, encoding it, and placing it in the SAID field, or digest field, which is the “d” field in this example:
The ‘H’ character is highlighted here to draw attention to the fact that is a special character. This special character is the type code in the CESR Master Code Table. This indicates the type of cryptographic algorithm being used, SHA3-256 in this case.
I see a problem…Those new to calculating and encoding SAIDs often encounter a problem here. If you take the raw Base64 encoded value of the JSON value {"d":"","first":"john","last":"doe"}
then you end up with the string value eyJkIjoiIiwiZmlyc3QiOiJqb2huIiwibGFzdCI6ImRvZSJ9
, which is nowhere close to the value shown in the picture of HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6
. Why are they different?
Doing a plain Base64 encoding of the JSON bytes misses an important step, the encoding step referred to above. The rest of the post dives deep into this encoding as it shows and explains how to construct a correct, CESR-encoded, SAID digest and explains the rationale behind why CESR encoding is designed the way it is.
Five parts of a SAID (SAID)?As mentioned earlier, a SAID is a cryptographic digest. Specifically, it is a kind of digest usable as a content addressable identifier, and it is embedded in the content it identifies. SAIDs were invented by Dr. Samuel Smith as a part of his work on key event receipt infrastructure (KERI), authentic chained data containers (ACDC), and composable event streaming representation (CESR).
To understand how SAIDs work you must learn the interplay of five different concepts including:
Bit boundaries – aligning on 24 bit boundaries using pre-padded bytes on the left/front of raw bytes Hash values – hashing input bytes with hashing functions to produce output hash values (digests) Encoding with the URL-safe variant of Base64 encoding, Using type codes to indicate type of hashing function and size of digest, The two-pass SAID calculation and embedding process.This article specifically covers SAIDs that are encoded in the CESR format. These CESR-style SAIDs
use pre-padding of pad bytes for bit padding to align on 24 bit boundaries, are compatible with a variety of common hashing functions, are encoded in the URL-safe variant of Base64 encoding (a.k.a. Base64URL), substitute type codes from the CESR Master code table (section 12.4.2) for ‘A’ front zero characters and are calculated from and embedded in the data they identify. How does it work? How are SAIDs calculated?The easiest way to understand a self addressing identifier is to create one. Starting with the JSON from above we walk through each of the five major concepts required to create a CESR encoded SAID.
7 Steps to Calculate and Embed a SAIDBriefly, the process is listed here. A detailed explanation and example follows this set of steps.
Get an object to calculate a SAID for with a digest field that will hold the SAID. In this case we use the JSON object below and the “d” field will hold the SAID. The field does not have to be empty though it can be. Prior to digest calculation it will be cleared and filled with the correct number of filler characters. Calculate the quantity of Base64 characters the final encoded bytes will take up and fill the digest field with that many ‘#’ characters. This value may be looked up from a parse table like the CESR Master Code Table based on the type of hashing function used. Replace the contents of the digest field, “d” in our case, with pound sign (“#”) characters up to the number of filler characters calculated in step 2. The calculated size and pad values used for this step are reused in step 4. Calculate a digest of the object with the filler ‘#’ characters added using the hash function selected. This will result in a quantity of digest bytes, specifically 32 bytes for the SHA3-256 algorithm. Calculate the quantity of pad bytes that when added to the digest bytes will give you a value length that is multiple of 24 bits. This math is shown below. For us this is 1 pad character giving us 33 bytes. This value may be looked up from a parse table like the CESR Master Code Table. Perform pre-padding by prepending the pad byte to the digest bytes to get padded raw bytes. Encode the padded raw bytes with the Base64 URL Safe alphabet. Pre-padding causes some characters at the start of the digest to be encoded as “A” characters which represent zero in the Base64 URL Safe alphabet. Substitute the type code for the correct number of “A” zero character(s) in the Base64 encoded string according to the CESR encoding rules from the CESR Master Code Table. Use the type code corresponding to the cryptographic hash algorithm used. In our case this is “H” because we are using the SHA3-256 algorithm. This is your SAID! Place the Base64 encoded, type code substituted string (your SAID!) into the digest field in your object. This makes your object self-addressing. 3 Steps to Verify a SAID Start with a SAID from an object you already have. Calculate the SAID for the object using the process shown above Compare the SAID you pulled out of the object with the SAID you calculated. If they match then the SAID verifies. Otherwise the SAID does not verify.An illustration will make clear why and how this process is done. Let’s walk through an example with a small JSON object. The concept applies to any size JSON object and objects of any serialization format such as CBOR, MessagePack, arbitrary text, or otherwise.
Example walkthrough with JSON and SHA3-256 Create Step 1: Get an object with some data and a digest fieldStarting with the JSON below we have a “d” field, or digest field, in which the SAID will eventually be placed. In our case it is empty though it could start with the SAID in the “d” field and the process would still work.
JSON being SAIDified:
{
"d": "",
"first": "john",
"last": "doe"
}
Create Step 2: Calculate the quantity of filler ‘#’ characters
The expected final size of the SAID must be known in advance in order to create a JSON object with a stable size. Calculating this quantity requires that you understand a major concept in CESR:
How to calculate pad sizes (quantity of pad bytes) and full sizes of values.Understanding this calculation will get you most of the way towards understanding another major CESR concept called “fully qualified Base64 representation” of a cryptographic primitive. A digest is a kind of cryptographic primitive.
Knowing the size in advance, and having it be stable, is critical for CESR’s type, length, value (TLV) encoding scheme. This stable size is achieved by filling the digest field with the same number of pound sign ‘#’ characters as the size of the SAID, which looks like this:
Correct number of filler characters added to digest field{
"d": "############################################",
"first": "john",
"last": "doe"
}
This enables the JSON to have the same size during and after the SAID calculation process, giving a stable size. In order to know the number of filler characters then you must calculate how many Base64 characters will be in the final SAID. Calculating how many Base64 characters are needed involves summing raw bytes and pad bytes needed to align on what is called a 24 bit boundary.
Final output has same size since Base64 characters count equals filler lengthAligning on this 24 bit boundary allows the final result with the SAID to have the same length as the version with the filler characters, 44 characters in our case:
{
"d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6",
"first": "john",
"last": "doe"
}
Remember when the “encoding” step was mentioned from above? That’s where this filler character and size calculation knowledge comes in. In this encoding step you learn about the CESR-style encoding using pre-padding, pre-conversion. Knowing how many filler characters to use requires understanding the concept of aligning on a 24 bit boundary. Aligning on a 24 bit boundary is where the pre-padding of CESR comes in. This calculation of pad bytes required to align on a 24 bit boundary is the primary difference between raw, or “naive”, Base64 encoding and CESR encoding.
First let’s delve into what a 24 bit boundary is, why it matters to Base64 encoded values, and then look at some diagrams that make Base64 post-padding and CESR pre-padding clear. In doing this we jump ahead a bit and show byte diagrams of the actual encoded digest since that will help introduce later steps.
24 bit boundary – from Base64The 24 bit boundary comes from the Base64 encoding format standard, RFC4648, specifically section 4. The reason a 24 bit boundary matters is because you can only use whole Base64 characters; there is no such thing as a fractional Base64 character. A Base64 character represents 6 bits of your raw bytes. A single byte is 8 bits. How do you reconcile the 6 bit Base64 character encoding to the 8 bits of your raw bytes? This is where a little math comes in, specifically the least common multiple.
Section 4 of the Base64 RFC 4648 describes the 24-bit groups that are the origin of the 24-bit boundary:
The encoding process represents 24-bit groups of input bits as output strings of 4 encoded characters. Proceeding from left to right, a 24-bit input group is formed by concatenating 3 8-bit input groups. These 24 bits are then treated as 4 concatenated 6-bit groups, each
RFC 4648 The Base16, Base32, and Base64 Data Encodings – Section 4
of which is translated into a single character in the base 64 alphabet.
Using these 24-bit groups ensures the value coming out of a Base64 decoder is the same value you put in. Separating raw bits into these 24 bit groups is where the phrase “aligning on 24 bit boundaries” comes from.
Splitting the 8-bit groups up into 6-bit groups requires a little math because 8 does not split evenly into 6. The math equation to do this is the least common multiple (LCM). LCM is used to determine the lowest number that both 8 and 6 divide into evenly, which is 24, thus the need for 24-bit groups, or 24-bit boundaries. Any value that is encoded into Base64 characters must be padded to reach a multiple of 24 bits. These 24-bit groupings allows you to cleanly convert all of your 8-bit bytes in to 6-bit Base64 characters and back to bytes without missing any bits.
Yet, if we have a stream that does not align on a 24 bit boundary then how do we create that alignment?
Pad characters on the END of a string are the answer to this in Base64.By adding the correct number of pad characters on the end of a Base64 stream then you always end up with a value aligned on a 24 bit boundary. The ‘=’ equals sign pad characters in a plain Base64 encoding indicate the quantity of pad bits that were used in the final Base64 character adjacent to the ‘=’ pad characters.
Pad bytes at the START of the raw bytes are the answer to this in CESR.By prepending the correct number of pad bytes on the start of a set of raw digest bytes then you always end up with a value aligned on a 24 bit boundary. Since the pad bytes are all zero bits then the resulting encoded value will start with one or more ‘A’ characters since they correspond to all zero bits in the Base64 alphabet.
Pad characters CalculationIn a plain Base64 encoding when encoding an array of bytes into Base64 that does not align on a 24 bit boundary the correct number of Base64 pad characters ‘=’ must be included. Why? Because in order to avoid data corruption in the decoded value you must know the precise original value, which means knowing how many pad characters to strip off and how many pad bits to strip out of the Base64 character adjacent to the padding. The decoder of your Base64 character needs to know how many bits of the last character used were just padding and how many were a part of your raw value.
You must signal the end of your raw bytes somehow. If, instead, you ignore, drop, or omit pad characters then you will confuse a Base64 decoder into thinking that pad bits were a part of your raw bytes, which you want to avoid because that will give you a different output value than what your input value was, meaning you would experience data corruption.
Pad characters must be included with a plain or “naïve” Base64 encoded value so that a Base64 decoder can strip the correct number of pad bits from the output giving you your original input bytes when decoding from Base64 characters to raw bytes. This is the purpose that Base64 pad characters serve. The pad characters indicate how many pad byes were used to encode a value in Base64.
CESR uses pad bytes and characters in a similar way, yet on the front, and with pre-conversion padding, so the rules for identifying and stripping pad bits are slightly different.
Yet, let’s stick with Base64 padding for now and come back to CESR padding later. If you are starting to get confused or lost then skip ahead to the diagrams below and come back to this explanation.
ASIDE – Calculating the quantity of Base64 pad characters based on input byte quantityFor a SHA3-256 digest this count is 44 characters. See the math below for an explanation. This number may also be found in the CESR Master Code Table for the type of algorithm used. Since we measure every raw value in terms of bytes (8 bits) then there are three possible scenarios, detailed here in the Base64 RFC, for the number of pad bytes required and thus pad characters.
A value ending with a single byte (8 bits) beyond a 24 bit boundary requires two bytes (16 bits) to meet a 24 bit boundary. This will have two ‘=’ pad characters. This means that your 8 raw bits + the 16 padding bits (two bytes) will equal 24 bits, aligning your raw value on a 24 bit boundary. A value ending with two bytes (16 bits) beyond a 24 bit boundary requires one byte (8 bits) to align on a 24 bit boundary. This will have one ‘=’ pad character. Take the 16 bits + one pad byte (8 bits) to get to 24 bits to align on the 24 bit boundary. A value ending with three bytes is already aligned on a 24 bit boundary (3 * 8 = 24)You can use the modulus operator ‘%’ to determine the number of ending bits you have. For 256 bits (32 bytes * 8 bits per byte) you end up with 16 bits, or two bytes, rule number two above. So we need the equivalent of one pad byte.
How Base64 handles pad bitsThe way that Base64 handles the need for pad bytes is to split the last byte into two characters, add zero bits to the last Base64 character, and then add the correct number of pad ‘=’ equals sign characters to the final output to end up with groups of 4 Base64 characters, which aligns on a 24 bit boundary because 4 * 6 bits per Base64 character = 24 bits.
What this means for a SAID – Calculating Pre-pad Bytes for CESRIn CESR padding is handled a bit differently because it repurposes the pad characters for type codes in its TLV encoding scheme. This means that what would have been zero bits representing ‘A’ characters in the Base64 encoded CESR value gets replaced with the type code, also called derivation code, in the final CESR value. To accomplish this CESR does pre-padding prior to conversion to Base64 characters. What this means for SAIDs is that all digest bytes must be padded at the front of the digest bytes to reach a multiple of 24 bits. Compare this to Base64 padding which occurs at the end of the digest bytes. Both scenarios are pictured below, Base64 padding and CESR padding.
Since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.
Now once you know the quantity of bytes that align on a 24 bit boundary you can do a simple calculation to get to the number of pad characters for your digest. Since 6 bits of every byte are put into a Base64 character (6 bit groups) then you can divide your total number of bits (264) by 6 to get the number of Base64 characters of your final digest.
264 (bits) / 6 (bits per Base64 char) = 44 (Base64 chars)
This means the total length of the resulting SAID will be 44 Base64 characters. So, you need 44 filler ‘#’ pound sign characters in your digest field of your JSON object prior to calculating the SAID.
Fixed width output – why is it needed?Consistent sizing of the resulting JSON object for stable size of the overall output is the primary reason for pad characters. In order to create the same size output both before and after the SAID is added into the JSON there must be an equivalently sized number of pound signs (44 in this case) placed into the same field where the SAID will go. This is used in CESR encoding because CESR data types are encoded with to a type, length, and value scheme (TLV scheme) that simplifies parsing. Size of the overall output is the length, or “L,” in TLV and it only works if you have a known width data.
{
"d": "############################################",
"first": "john",
"last": "doe"
}
Now that you know the rules for calculating the number of pad characters then we are ready to illustrate the calculation process with diagrams.
Diagram for plain “naïve” Base64 encoding of SHA3-256 digestBase64 uses post-padding, post-conversion of pad characters, as shown in the diagram below. You start with the raw digest. All the boxes in this diagram represent the raw bytes of the digest. There is no padding yet because the value is raw and is not yet converted to Base64 characters.
Binary bits of 32 byte SHA3-256 digest of above JSON with ‘#’ fillerFor those following along in code the raw bytes of the 32 byte SHA3-256 digest of the JSON above (with the ‘#’ filler characters) are represented in binary as follows:
1111001001011011010101100010111010011111011001101111000110001101000010000000010010000011100010110000000000000001100111110110110000101001010000110100100101001000111110110110011100010001110100110010011010101000010001000100101011100100000011111110100011111010
Take a look at the last two bytes are 11101000
and 11111010
. This factors in to the last two characters adjacent to the pad character as you see below.
Encode this 32 byte digest to Base64 URL Safe and you get get:
What happened here is that four bits (1010
) of the last byte ( 11111010
) were encoded into the last character lowercase ‘o’ adjacent to the pad character. If you look at the value for lowercase o in the Base64 alphabet you will see that lowercase ‘o’ has the bit pattern 101000
. Yet it only pulled four bits from the last byte of 11111010
so where did the last two bits (00
) come from? They were added in by the Base64 encoder. These two pad bits are why the corresponding final value has a single equals sign ‘=’ pad character. That instructs the Base64 encoder to strip two bits from the last character during the decoding process:
IMPORTANT: Base64 does not add the padding to the raw bytes prior to conversion. Instead it adds the padding while converting the 6 bit groups of the raw bytes into Base64 characters.
Due to the fact that 32 bytes, 256 bits, does not evenly align on a 24 bit boundary, is not a multiple of 24, the Base64 encoder splits the last byte into two different Base64 characters since 8 bits does not evenly fit in one 6 bit group and must be spread across two 6-bit groups. Each of these 6 bit groups each get their own Base64 character. In this case, the last two bytes 11101000
and 11111010
get spread across the last two characters ‘P’ (001111
) and ‘o’ (101000
).
Because of how the math works when splitting the 8-bit byte groups into 6-bit Base64 character groups the ‘o’ character got four bits from the very end of the digest. Yet four bits is not enough for a Base64 character so the Base64 encoder adds two zero bits on the end, signified with white boxes containing zeroes. Before the pad character is added then we are at 43 Base64 characters (6 bit groups, 258 bits), which is not a multiple of 24 bits. When the pad character ‘=’ is added then we get to a 44 characters (264), which is a multiple of 24 bits, meaning the encoding completed successfully.
Base64 Encoded SHA3-256 DigestWith the fully padded value you end up with a valid, encoded, Base64 value that looks like the following bit diagram:
The C2 character at the end shares some bits with the raw bytes of the digest and also contains some padding zero bits. The last character, C1, is an equals sign ‘=’ pad character. The fact that there is one pad character indicates to the Base64 decoder that there are two zeroed pad bits to remove from the last character, ‘C2’, during decoding in order to get back to the original digest bytes.
‘=’ is wasted space?You could consider the pad characters ‘=’ as wasted space that could be useful if repurposed. All of the pad bits used for the equals sign could represent something. This is exactly what CESR does except it moves the padding to the front of the bytes so that it can have a uniform TLV encoding format. TLV encoding formats require the type character to be at the front of the value, so using post-padding like Base64 does would not work.
Along these same lines, SAIDs do not use Base64-style padding because it does not enable separability of individual concatenated values due to the fact that there is no easy way to cleanly and reliably separate individual values out of a Base64 encoded stream of bytes. The CESR specification introduction mentions this:
This Composability property enables the round-trip conversion en-masse of concatenated Primitives between the text domain and binary domain while maintaining the separability of individual Primitives.
Composable Event Streaming Representation ToIP specification – Dr. Sam Smith
Now that you understand how the plain or “naïve” Base64 encoding works then we turn our attention to CESR style pre-padding.
CESR Byte Padding: Pre-padding, Pre-conversionIn CESR the padding of values occurs with the raw bytes prior to encoding to Base64 as shown below in the white box containing ‘B33.’
What this means is that the raw value, prior to conversion, already aligns on a 24 bit boundary. Due to this alignment pre-conversion then there will never be any Base64 pad characters ‘=’ in the output.
How many bytes to prepend?How do you know how many bytes to prepend? With a similar calculation we did above to find the number of filler characters
Since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.
Again, once you know the quantity of bytes that align on a 24 bit boundary you can do a simple calculation to get to the number of pad characters for your digest. Since 6 bits of every byte are put into a Base64 character (6 bit groups) then you can divide your total number of bits (264) by 6 to get the number of Base64 characters of your final digest.
264 (bits) / 6 (bits per Base64 character) = 44 (Base64 Characters)
So 44 will be the quantity of filler characters to put into the JSON object in order to calculate a SAID.
What happens when prepending bytes for CESR style encodings?When encoding a value that requires padding with CESR-style padding (up front), instead of ‘=’ at the end like Base64-style padding would produce you end up with ‘A’ characters on the front of your encoded value. You also end up with the one character adjacent to the ‘A’ character(s) including some pad bits and some raw bits, as shown below in the bit diagram.
The intermediate encoded value looks like the below value that is not yet a SAID. This is not yet a SAID because the ‘A’ character has not yet been replaced with a type code from the TLV scheme indicating this is a SHA3-256 digest.
This ‘A’ character represents all zero bits (000000
) in the Base64 alphabet.
In binary the full, pre-padded digest value (all 33 bytes) looks like the following. Notice the zero bits at the front.
000000001111001001011011010101100010111010011111011001101111000110001101000010000000010010000011100010110000000000000001100111110110110000101001010000110100100101001000111110110110011100010001110100110010011010101000010001000100101011100100000011111110100011111010
The first two bytes are 00000000
and 11110010
which get encoded into Base64 as shown below. Six of the zero pad bits get encoded as an ‘A’ character and two of the zero pad bits get included in the capital ‘P’ character which also has four bits from the next raw byte of data from the digest.
This diagram illustrates how CESR does pre-padding with pad bytes of zero bits prior to performing a Base64 encoding on the fully padded raw value. The next diagram of characters shows the space a fully padded, encoded, CESR-style value would look like.
As you can see, the padding is at the front of the encoded value rather than the back like Base64 does. And the character with shared pad and raw bits is adjacent to the pad character at the front of the Base64 encoded value.
To get to the final SAID then you replace the ‘A’ character with the appropriate type code, or derivation code, yet we are getting ahead of ourselves a bit too much. Let’s now get into the calculation of the digest.
This step showed you how to calculate the appropriate number of filler ‘#’ pound sign characters to put in to the digest field in your JSON object. The next step shows you how to calculate a digest of that JSON object.
Creation Step 3: Calculate a digest of the dataWhen calculating a digest then you take the data with the correct number of filler characters added to the digest field and you simply take a digest of it. So in our case we would take a digest of the following
{
"d": "############################################",
"first": "john",
"last": "doe"
}
In Python taking a digest of this data would be as simple as the following:
import hashlib raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha3_256(raw_value).digest() # hash function ↑↑↑↑This is a simple step and is very similar for any other algorithm such as SHA2-256, Blake3-256 or otherwise. You use the desired type of hash function.
The only other thing to be aware of here is that if you create a digest that is sized differently than 32 bytes, such as a SHA3-512 digest (64 bytes) then you need to also change the number of pad bytes, which gets into the next step.
Creation Step 4: Calculate the quantity of pad bytesThe calculation for the quantity of pad bytes is very similar to the calculation for the quantity of filler ‘#’ characters needed in Step 2. In fact, it is a subset of that calculation. The goal with pad characters is to make sure that the final value aligns on a 24 bit boundary as mentioned above.
For example, since the SHA3-256 digest we start with is 32 bytes, or 256 bits (not a multiple of 24), then all we need to add is one byte to get to 264 bits, which is a multiple of 24, or 33 bytes.
Deeper into Modulus Math for Pad BytesTo get a bit deeper into the math, one way to do this calculation with the modulus operator is to find out how many characters are necessary to completely fill a 3 byte group. Since 3-byte groups are multiples of 24 then you can use a modulus calculation to see how far away you are from filling a three byte group by doing a modulus 3 operation in two steps:
Step 1: take bytes mod 3
32 bytes mod 3 = 2 (bytes)
meaning there are two bytes already in the last group of three (24 bit boundary).
Step 2: subtract bytes in group from group size
So to see how many bytes you must add to get to the 24 bit boundary (3 byte group) you subtract the quantity of bytes you have from the group size:
3 (group size) – 2 (bytes in group) = 1 (pad bytes needed to fill group)
Due to how modulus arithmetic works you will only ever have three possible values from this equation:
3 – (bytes mod 3) = 0 (pad bytes) 3 – (bytes mod 3) = 1 (pad bytes) 3 – (bytes mod 3) = 2 (pad bytes)You never have to worry about three pad bytes because that would be an even multiple of 24 which means your raw value would already align on a 24 bit boundary and thus not need any pad bytes.
So, to review, for us the calculation of (3 - (32 mod 3)) = 1 pad byte
gives us a single pad byte to be prepended to our raw value, as shown below in the ‘B33’ box.
As mentioned before, CESR does pre-padding, pre-conversion which means that the pad byte we found we need is added to the front of the array of raw bytes for the SHA3-256 digest. The next step covers encoding this padded raw value.
Creation Step 5: Base64 URL Safe Encode the padded raw bytesNow that the raw value from Step 4 is properly padded then you encode it with Base64 URL Safe encoding. CESR uses Base64 URL Safe encoding rather than plain Base64 encoding so that CESR values can safely be used in URLs and filenames.
import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d": "############################################", "first": "john", "last": "doe"}' digest = hashlib.sha3_256(raw_value).digest() padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) # encode to base64 ↑↑↑↑ assert encoded == b'APJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6' assert len(encoded) == 44Now that you have the Base64 URL Safe encoded value then you are ready to finish off this SAID creation by replacing the ‘A’ pad character at the front of the encoded value with the appropriate value from the CESR Master Code Table.
Creation Step 6: Substitute Type Code for the front ‘A’ character(s)When CESR pre-padded the raw value to get to a 24 bit boundary the purpose of that was to be able to repurpose the wasted space of the pad character for a type code in CESR’s TLV encoding scheme. The ‘A’ character at the front of the value in this scheme is considered to be a pad character. This pad ‘A’ character will be replaced with the appropriate type code, or derivation code in CESR parlance, from the CESR Master Code Table.
For a SHA3-256 digest that type code is ‘H’ as seen in the following subset of the CESR Master Code Table.
The substitution gives us a final value of HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6
as seen in the following substitution diagram.
The substitution of the ‘A’ character with the ‘H’ character is the final part of what is called CESR encoding a raw digest value into a CESR-style self addressing identifier. This SAID is a front-padded, Base64 encoded, and type-code substituted, string of Base64 characters.
The final value can be created by the code as follows:
import hashlib from base64 import urlsafe_b64encode raw_value = b'{"d":"############################################","first":"john","last":"doe"}' digest = hashlib.sha3_256(raw_value).digest() padded_digest = b'\x00' + digest encoded = urlsafe_b64encode(padded_digest) b64_str_list = list(encoded.decode()) # convert bytes to string of chars for easy replacement of 'A' b64_str_list[0] = 'H' # replace first 'A' character with 'H' type code b64_str = ''.join(b64_str_list) # convert string of chars to string with .join() assert b64_str == 'HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6' assert len(b64_str) == 44 # length should still be 44 characters, 264 base64 bits, a multiple of 24 bits Creation Step 7: Place the Front-Padded, Base64 encoded, Type-code Substituted string in the digest fieldNow we can take this correctly padded, CESR encoded value and place it into the digest field in our JSON object, replacing the filler ‘#’ characters with the final, valid SAID:
{
"d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6",
"first": "john",
"last": "doe"
}
This takes us back to where we started off, with a valid SAID and a SAIDified JSON object.
What about verification?What is nice about verification is that it is as simple as calculating the SAID again of a JSON object and comparing that to a SAID you are handed.
Verification Step 1: Start with a SAID from the object you already haveSay you are starting with the below object that has already had a SAID calculated and embedded in the digest field, the “d” field here.
{
"d": "HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6",
"first": "john",
"last": "doe"
}
To get the SAID from this object you extract the value of the “d” field, giving you HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6
Verification is easy because all you need to do is take steps 1 through 6 above and re-calculate the SAID on the JSON object provided. Once you have recalculated the SAID, which will be HPJbVi6fZvGNCASDiwABn2wpQ0lI-2cR0yaoRErkD-j6
again, you can perform the comparison in step 3.
If the SAID the object started with matches the SAID you calculated from the object then you know the object has not been changed and that the SAID is valid. Otherwise either your SAID is invalid or the object has changed.
Review Calculating a SAIDNow you understand how we SAIDify a JSON object by doing the following seven step process:
Start with a JSON object we want to add a SAID to that has a digest field. Calculate the quantity of Base64 characters the final, pre-padded, encoded raw digest bytes (SAID) will take up and fill the digest field with that many ‘#’ characters. Calculate a digest of the bytes of the JSON object after the ‘#’ filler characters are added. Calculate the quantity of pad bytes needed to align on a 24 bit boundary and prepend that to the raw bytes for a digest. Encode the padded raw bytes with the Base64URLSafe alphabet. Substitute the appropriate type code in place of the ‘A’ character(s) at the front of the encoded string. This final value is your SAID Place the final SAID value into the digest field of your JSON object.Pre-padding and type code substitution prior to Base64 encoding is the essence of CESR-style self addressing identifiers. The steps above may seem overwhelming at first, though once you mentally anchor in that CESR pads at the start and that padding gives you ‘A’ characters you can reuse for type codes then you have mastered the fundamentals of what makes CESR style SAIDs work.
Verifying a SAIDVerification of a SAID is easy because you just calculate it again from the original JSON object, or other data object you are using. If the SAIDs match then it verifies; if they don’t then the data changed.
Extra Learning Alert – fully qualified Base64 primitiveAnd, as a nice side note, you happen to now know what the phrase “fully qualified base64 primitives” in KERIpy means. All that means is that your encoded value has been pre-padded, pre-conversion, and has had its type code added to the front, as we did here with substitution, with the exception that some CESR primitives
Give me a library please! I don’t want to manage these detailsIn case this article has convinced you that you do not ever again want to worry about the vagaries of aligning on 24 bit boundaries for Base64 or CESR values then you are in luck. There are multiple implementations of the SAID process that can meet your needs in a variety of different languages.
The Python reference implementation in Web Of Trust’s KERIpy’s Saider.saidify. The Human Colossus Foundation’s Rust implementation with WASM bindings for their JavaScript package. See their cool SAID generator and verifier demo here where you can try a whole list of different algorithms. SAIDify, my own Typescript implementation of the SAID creation process. Implementations Web Of Trust KERIpy PythonThe Python example below from KERIpy shows a unit test showing the usage of the KERIpy Saider.saidify library code to calculate a SAID. The SAID is stored in the .qb64
property of Saider
. The term qb64
stands for “qualified base64” which means a left-padded, Base64 encoded, type code substituted value as described above.
Start with their cool demo site of generating and verifying SAIDs:
If you want to dive into their code the linked test basic_derive_test
shows the Rust code for the cool SAD
macro #[derive(SAD, Serialize)]
that can turn any Rust struct along with the #[said]
field attribute for the SAID digest field into a self-verifying data structure.
use said::derivation::HashFunctionCode;
use said::sad::SAD;
use said::version::format::SerializationFormats;
use said::SelfAddressingIdentifier;
use serde::Serialize;
#[test]
pub fn basic_derive_test() {
#[derive(SAD, Serialize)]
struct Something {
pub text: String,
#[said]
pub d: Option<SelfAddressingIdentifier>,
}
let mut something = Something {
text: "Hello world".to_string(),
d: None,
};
let code = HashFunctionCode::Blake3_256;
let format = SerializationFormats::JSON;
something.compute_digest(&code, &format);
let computed_digest = something.d.as_ref();
let derivation_data = something.derivation_data(&code, &format);
assert_eq!(
format!(
r#"{{"text":"Hello world","d":"{}"}}"#,
"############################################"
),
String::from_utf8(derivation_data.clone()).unwrap()
);
assert_eq!(
computed_digest,
Some(
&"EF-7wdNGXqgO4aoVxRpdWELCx_MkMMjx7aKg9sqzjKwI"
.parse()
.unwrap()
)
);
assert!(something
.d
.as_ref()
.unwrap()
.verify_binding(&something.derivation_data(&code, &format)));
}
SAIDify
If you want to use a Typescript library that is about 530 lines of code you can go with my SAIDify library. The below example shows how to use the library with Typescript.
Start with an NPM install
npm install saidify
And then you can use the saidify(data, label)
function to SAIDify any JavaScript object you have as long as you indicate which field is the digest field, the “label” field, which defaults to the “d” field.
The key takeaways from calculating SAIDs are:
Use pre-padded bytes to align on a 24 bit boundary prior to encoding as Base64 characters. Substitute type codes in for the leading ‘A’ character(s) of a SAID. It is easy to chose different algorithms for the SAID calculation process. Just make sure you use a code on the CESR Master Code Table if you want to be CESR compliant. There are multiple implementations of the SAID algorithm you can use.Now go make some SAIDs!
References: HCF oca-spec #58 RFC 4648: The Base16, Base32, and Base64 Data Encodings, specifically section 5 Composable Event Streaming Representation (CESR) ToIP Specification, specifically section 12.6 Self Addressing Identifier IETF draft specification SADs, SAIDs, and ACDCs video presentation by Daniel HardmanOne of the problems I have encountered in trying to do Retrieval Augmented Generation (RAG) where a complete single document was not uploaded for queries on that document alone is that chunks created from documents for embedding often lose context. I have often considered what would be the outcome if one were to put some limited but useful context along with each chunk. Now I know. Introducing Anthropic's contextual retrieval concept.
One of my concerns with placing some additional context for each chunk is that you’d probably need to pass the entire document being chunked as context along with each individual chunk. That would be very expensive and slow. Now, with Anthropic’s Claude prompt caching, the cost and latency is reduced significantly and it seems actually doable for chunk contexting (is contexting a word?). An initial prompt including the full document could be done with that prompt cached for future prompt reference.
I plan to try this out.
Check out the Anthropic “Introducing Contextual Retrieval” post for greater details.
"Beyond the ability to fine-tune open models for focused applications, Kal’tsit says, another advantage of local models is privacy. Sending personally identifiable data to a commercial service could run foul of data-protection regulations. “If an audit were to happen and you show them you’re using ChatGPT, the situation could become pretty nasty,” she says."
Many organizations have similar privacy needs to these researchers, who simply can't send confidential patient data to third party services run by vendors like OpenAI. Running models locally - either directly on researcher laptops, or on researcher-controlled infrastructure - is inevitably going to be a big part of how AI is used in any sensitive context.
We have the same needs at ProPublica - unless journalists are examining public data, they shouldn't use hosted services like ChatGPT that might leak identifying information about sources, for example. Local models are a huge part of the future for us, too.
[Link]
On a recent trip I saw this pair of Latin phrases tattooed on the back of a flight attendant’s arms:
Left: Deo absente. Right: Deum culpa.
I took Latin in middle school, and could guess what the combination might mean. It’s not a common construction, and a search seems to confirm my guess. Both Google and Bing take you to a couple of Reddit posts in r/Latin.
Would this be the correct translation?
A song I like, Deus in absentia by Ghost, has that line in it intending to mean “In the absence of God”, so I was looking into alternate translations/syntax of the phrase intending to mean “In the absence of God; Blame/Fault God”. Would this make sense: “Deum in absente; Culpa Deus” or “Deus Culpa”?
Does the phrase “Deus In Absentia, Deus Culpa” make sense?
I’m using this for a tattoo and want to be absolutely sure it works in the sense of ‘In the absence of God, blame God’. All help appreciated!
Is that the same person I saw? If so, the responses in r/Latin seem to have guided them to the final text inked on their arms. And if so, the message is essentially what I had guessed. The intent of the message, though, is open to interpretation. I’m not quite sure how to take it. What do you think it means? Would it have been rude to ask?
こんにちは、富士榮です。
LINEログインやLIFFのロードマップが更新されていますね。
https://developers.line.biz/ja/docs/line-login/roadmap/
ログイン回りをピックアップしておきましょう。
リリース時期機能概要対象プロダクト2024年後半LINEログインのダークモード表示の対象拡大LINEログインにおいて、ダークモード表示に対応するページの範囲を拡大します。LINEログイン時期未定LINEログイン v1、v2.0へのアクセス遮断既に廃止されているLINEログイン v1、および非推奨となっているLINEログイン v2.0へのアクセスを完全に遮断します。LINEログイン継続実施アクセシビリティ対応LINEログインを、より多くのエンドユーザーにとって使いやすいプロダクトにするため、アクセシビリティ対応を継続して推進していきます。LINEログイン継続実施同意画面をより使いやすいものに改善LINEログインを、より多くのエンドユーザーにとって使いやすいプロダクトにするため、A/Bテストなどの分析に基づき、機能やUIの改善を継続して推進していきます。LINEログインまぁ、見た目の話が多いですが、旧バージョンのAPIが遮断される時期は引き続きみておかないといけませんね。(少なくとも1.0はもう使われていませんが、2.0からは結構使われているので)
Powerpipe dashboards can now connect not only to Steampipe but also to SQLite and DuckDB. This creates a combinatorial explosion of possibilities, including dashboards that use SQL to visualize large datasets read from Parquet files by DuckDB.
SQL Translation From Postgres to SQLite and DuckDB
Part of the LLM series at The New Stack.
"Microsoft and Constellation Energy have announced a deal that would re-open Pennsylvania's shuttered Three Mile Island nuclear plant. The agreement would let Microsoft purchase the entirety of the plant's roughly 835 megawatts of energy generation—enough to power approximately 800,000 homes—for a span of 20 years starting in 2028, pending regulatory approval."
This seems to be the new front in datacenter technology: purchasing or building entire nuclear plants in order to cover the energy cost. It is significantly better than high-emissions power from sources like coal, but it also speaks to the increased demand that new technologies like AI represent.
As ArsTechnica points out:
"Industry-wide, data centers demanded upward of 350 TWh of power in 2024, according to a Bloomberg analysis, up substantially from about 100 TWh in 2012. An IEA report expects those data center power needs to continue to rise in the near future, hitting the 620 to 1,050 TWh range by 2026."
AI is a huge and growing part of that, although let's not pretend that the internet industry overall has low emissions. We often pretend we're greener than we are, simply because we can't directly see the output - but there's a lot of work to do, and a lot of carbon footprint to own up to.
[Link]
"Links — connections between ideas — are the magic system of the Internet. They power the open web, enriching online writing. Generative AI is the parasitic dark magic counterpart to the link."
I love Tracy's observation that "online, we think together", which also calls back to the original definition of the word blog ("weblog" = "we blog").
Links are context, further thought, community. Removing that context removes depth. They're inherent to the web: they're what the web is. When platforms want to strip-mine value from our work - our writing, our thinking - by lifting it away from its community and context, we need to fight back. And fight back we will.
[Link]
[Chris Metinko at Crunchbase News]
"Last year, venture funding to Black-founded U.S. startups cratered — totaling only $699 million and marking the first time since 2016 that the figure failed to even reach $1 billion, Crunchbase data shows."
And:
"While last year did not see Black founders raise $1 billion in total, this year such founders and startups are on pace to raise less than even half-a-billion dollars. In fact, the combined total of funding to Black founders in the second half of last year and the first half this year is only $351 million."
While some of this is a reflection of the ongoing tightening in VC overall, that certainly doesn't account for a pull-back of this magnitude.
VC is often a connections-based business: investors like to have warm introductions from people they trust. It helps to be part of the in-group, and given the demographics and backgrounds of most investors, Black founders may be excluded. Open calls for pitches help, but the single biggest thing venture teams could to do widen their net and make sure they don't miss out on talented Black founders is for their own teams to be more representative. This article doesn't directly mention whether there's been progress on that front - but the numbers suggest maybe not.
[Link]
PSA for anyone who switched to Arc as their main browser (hey, that's me!): it had a giant vulnerability that the team, at the time of writing, doesn't seem to have acknowledged publicly, although it has been patched.
Aside from the lack of disclosure, perhaps the biggest ongoing concern for me is in the last few paragraphs:
"while researching, i saw some data being sent over to the server [...] this is against arc's privacy policy which clearly states arc does not know which sites you visit."
Sigh.
[Link]
こんにちは、富士榮です。
MicrosoftからMVP(Most Valuable Professional)というアワードをいただき初めて今年で15年目、という話を前に書きましたが、最近この手のアワードやIT系の資格試験などではOpenBadge形式で資格証明がもらえることがあります。
これまで何度かイベントではOpenBadgeの中身について話をしてきましたが、要するにPNGやSVGファイルの中身(PNGならiTXt領域、SVGならopenbadge属性)へJSON-LDで記述されたクレデンシャルの情報を埋め込んでいます(OpenBadge 2.0、Hosted形式の場合)。
この埋め込み作業のことをBake(ベイク)と言い、こちらに技術仕様が公開されています。
https://www.imsglobal.org/sites/default/files/Badges/OBv2p0Final/baking/index.html
なお、最近はOpenBadge 3.0がW3C Verifiable Credentials Data Model 2.0を使って定義されようとしています。(Finalizeと言いつつVCDM2.0がFinalにならないのでどうするつもりなんだ、という話もありますが)
OpenBadgeにはHosted型とSigned型の2種類が定義されており、これまでは検証時に発行者のURLへ問い合わせを行うHosted型が中心でしたが、OpenBadge 3.0からはVerifiable Credentialsを利用することでSigned型(バッジ単体で検証ができる)が中心になってくると思われます。
ただ、現状はMVPバッジはCredly社(IT系の資格の受験をするときにお世話になった方も多いであろうピアソン社を数年前に買収した会社で民間の資格証明の最大手ですね)が発行するOpenBadge 2.0、Hosted型のバッジです。
こんな感じでCredlyのバッジポータルでバッジの確認やダウンロードができます。
ちなみに実際のOpenBadgeイメージはこちらから取得できますので、こうやってブログやWebサイトで他の方へ提示(配布)することもできるわけです。
先ほど書いた通りiTXt領域にJSON-LD形式で埋め込まれているわけですが、IMS Globalがこちらで検証サイトを公開しているので、こちらを使いましょう。
https://openbadgesvalidator.imsglobal.org/
このサイトを使って取り出したOpenBadgeがこちらです。
{ "input": { "value": "https://www.credly.com/api/v1/obi/v2/badge_assertions/8793fdbf-80a5-4240-977e-12ac45574df3", "input_type": "url" }, "graph": [ { "@context": "https://w3id.org/openbadges/v2", "type": "Assertion", "issuedOn": "2024-09-17T00:00:00.000Z", "recipient": { "type": "email", "identity": "sha256$071281e466032326ffe4c3238545d31970b2b61d75fae181a283ac0aace09264", "hashed": true }, "badge": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49/badge_classes/2b797f06-fdd4-4ec7-b13c-e8f79915da0c", "verification": { "type": "HostedBadge" }, "evidence": [], "id": "https://www.credly.com/api/v1/obi/v2/badge_assertions/8793fdbf-80a5-4240-977e-12ac45574df3" }, { "@context": "https://w3id.org/openbadges/v2", "tags": [ "Community", "Leadership", "Technology" ], "name": "2024 Microsoft Most Valuable Professional (MVP)", "image": "https://images.credly.com/images/9e9359a4-fe7e-4e02-8eb0-6c2b7947345a/image.png", "alignment": [], "criteria": { "narrative": "Award: Individuals must have deep knowledge and expertise aligned to a Microsoft product or service or related open-source technologies. Through community work, qualifying applicants should be able to demonstrate their technical expertise. https://mvp.microsoft.com/", "id": "https://www.credly.com/org/microsoft-student-programs/badge/2024-microsoft-most-valuable-professional-mvp" }, "id": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49/badge_classes/2b797f06-fdd4-4ec7-b13c-e8f79915da0c", "type": "BadgeClass", "description": "The Microsoft MVP Program recognizes outstanding members of technical communities for their community participation and willingness to help others. Above all else, it is a people-powered program, made up of individuals whose passionate commitment to innovation has made its dynamic growth possible.", "issuer": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49" }, { "@context": "https://w3id.org/openbadges/v2", "type": "Issuer", "id": "https://www.credly.com/api/v1/obi/v2/issuers/214390fb-07bc-4575-80f2-f2c325f71c49", "name": "Microsoft MVP and Student Ambassadors Communities", "image": "https://images.credly.com/images/8f11d73e-9c33-4788-a602-88c761957c90/blob.png", "description": "The Microsoft MVP and Student Ambassadors Programs are communities where student experts, technology professionals or industry leaders transform their passions and skills into globally recognized community leadership.", "email": "info@credly.com", "url": "https://mvp.microsoft.com/" } ], "report": { "valid": true, "messages": [], "warningCount": 0, "validationSubject": "https://www.credly.com/api/v1/obi/v2/badge_assertions/8793fdbf-80a5-4240-977e-12ac45574df3", "openBadgesVersion": "2.0", "errorCount": 0 } } このとおり、IssuerはCredly社となっており、マイクロソフトが直接発行しているわけではないことがわかります。 また、OpenBadgeの特徴としてあくまで資格情報を表現するものとなっていることが挙げられます。何を言っているかというとバッジを提示する人とバッジが指し示す人が一致していることは表現していないということです。(私が他人のバッジを保持して提示することも可能。まぁ単なる画像ファイルですから当然ですね) そのため、Verifyをする際にバッジを提示する主体とバッジが指し示す主体が一致していることを検証サイト側で行うことが必要となります。
The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.
Today, we cover:
Industry pulse. OpenAI’s new model, LinkledIn opts everyone into GenAI training (save for those in the EU), Unity reverses runtime fees, what a startup “fire sale” looks like for employees, and more.
What does Amazon’s 5-day RTO mean for tech? The online retailer is the first Big Tech giant to mandate a strict, 5 days in the office policy. It says it wants things back exactly like they were before the pandemic. One reason is likely something less discussed: tax incentives from cities granted for staff being onsite, daily
Tech debt: further learnings. Two weeks ago, Lou Franco shared hard-earned lessons on keeping tech debt at bay. Lou shares two additional learnings: using tech debt payments to get into the flow, and how big rewrites need heavyweight support.
1. Industry pulse OpenAI’s new model: o1Last week, OpenAI revealed its latest model called OpenAI o1. Until now, LLM models got bigger in how much data they were trained on, but o1 is a change of direction. It spends more time “thinking” before it responds: basically, it chains several LLM operations one after the other. As software engineer Simon Willison puts it:
“One way to think about these new models is as a specialized extension of the chain of thought prompting pattern — the “think step by step” trick that we’ve been exploring as a a community for a couple of years now, first introduced in the paper Large Language Models are Zero-Shot Reasoners in May 2022.”
The new model is far more capable in its responses – even though these responses take more time to process – and brings a new dimension in how to use language models. Because the model spends more compute time on answers, it has become more expensive to use: o1 costs 100x as much as GPT-4o mini, and 3-6x as much as GPT-4.
This approach feels like the biggest recent development in LLMs this year. It’s worth withholding judgment until we use it more, and learn more about its capabilities and tradeoffs. AI keeps moving fast, and OpenAI is at the forefront.
LinkedIn opts everyone into GenAI training, except in EU"A new Federal Trade Commission staff report that examines the data collection and use practices of major social media and video streaming services shows they engaged in vast surveillance of consumers in order to monetize their personal information while failing to adequately protect users online, especially children and teens."
None of this is particularly surprising, but it's frankly nice to see the FTC see it and recommend taking action. Lina Khan is doing great work actually holding software monopolies to task.
My favorite recommendation is the first one:
"Congress should pass comprehensive federal privacy legislation to limit surveillance, address baseline protections, and grant consumers data rights;"
This should have happened years ago, and even now, getting it done will be a struggle.
This one, on the other hand, falls into the "and pigs should fly" category:
"Companies should not collect sensitive information through privacy-invasive ad tracking technologies;"
Yes, companies should not, but they will until comprehensive privacy legislation is enacted with meaningful penalties. This report is a step in the right direction; that legislation must come next.
[Link]
"The creator of an open source project that scraped the internet to determine the ever-changing popularity of different words in human language usage says that they are sunsetting the project because generative AI spam has poisoned the internet to a level where the project no longer has any utility."
Robyn Speer, who created the project, went so far as to say that she doesn't think "anyone has reliable information about post-2021 language used by humans." That's a big statement about the state of the web. While spam was always present, it was easier to identify and silo; AI has rendered spam unfilterable.
She no longer wants to be part of the industry at all:
"“I don't want to work on anything that could be confused with generative AI, or that could benefit generative AI,” she wrote. “OpenAI and Google can collect their own damn data. I hope they have to pay a very high price for it, and I hope they're constantly cursing the mess that they made themselves.”"
It's a relatable sentiment.
[Link]
I came across a great AI Retrieval Augmented Generation resource.
It is a Github repo: Advanced RAG Techniques: Elevating Your Retrieval-Augmented Generation Systems.I’ll just copy and paste their introduction here.
“Welcome to one of the most comprehensive and dynamic collections of Retrieval-Augmented Generation (RAG) tutorials available today. This repository serves as a hub for cutting-edge techniques aimed at enhancing the accuracy, efficiency, and contextual richness of RAG systems.”
All I can say is, wow. It really covers a lot of ground. I plan to dig into it and will report back.
こんにちは、富士榮です。
SAMLaiです。AIではありません。愛です。サムライです。
ということで、以前アナウンスさせていただいた大学ICT推進協議会(AXIES)の認証基盤部会主催の勉強会でSAMLを知っている人向けのOpenID Connect講座をやってきましたので資料をこちらにおいておきます。
もちろん一般企業などにも使える話だと思うので、ご覧ください。
[Myoung-Gi Chon in The Conversation]
"We found a disturbing link between work-related communication outside of regular hours and increased employee burnout. Answering emails after hours was linked to worse productivity, employees badmouthing their employers and other negative behaviors."
This is an important (if perhaps obvious) finding, but it's worth diving a little deeper and asking follow-on questions. Is it just the act of sending communications out of working hours? Or is it also an underlying organizational culture of disrespect for employees that allows such a thing to be normal?
The reason I ask is that one might be tempted to address the symptom - those out of hours emails - when there's likely something deeper to also take care of.
In the same vein, that's not to say that you shouldn't address the expectation of ubiquitous availability because the larger cultural work is still to be done. They clearly are bad in themselves, and do lead to exhaustion and burnout. But it seems to me that you have to do the bigger work, too.
[Link]
"Those of us who are makers, who create the source, need to be wary of those who would take our creations and squeeze out the juice. They’re grifters who will hop onto the next fad, but we’re trying to build something big here, something long term—something that lasts for generations."
Matt Mullenweg takes a strong stand for open source, and against companies that claim to be open but aren't quite.
Of course, not everything Automattic does is open source - its commercial operations were kicked off by the centralized Akismet anti-spam service, after all - but I agree that this clarity is useful.
It ends with a call to action: to support organizations that support ecosystems rather than abuse them. It's hard to disagree with that.
[Link]
Hi, this is Gergely with a free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of software engineers and engineering managers. To get weekly articles in your inbox, subscribe:
This publication has turned three years old, which feels like the right time to pause and reflect on the past 12 months for The Pragmatic Engineer.
At time of publication, 759,402 readers subscribe to this newsletter. This is 300,000 readers up on a year ago, when it was just under 461,000. This figure far exceeds the size of the audience I assumed would be interested in software engineering deep dives into some fascinating and challenging topics, when I launched this publication. Thank you for your trust!
I’d like to extend a very special thanks to all paying subscribers: the publication runs on your support, and it enables us to deliver well-researched deep dives. Many paying subscribers expense this newsletter from their company learning and development budget. If you have such a budget, here’s an email to send to your manager. There’s also reduced prices for people in countries with lower average income than the US and Western Europe, and student discounts. If you enjoy the newsletter, you can also gift a subscription to others.
Speaking of deep dives: over the past year full subscribers received two articles almost every week: a deepdive on Tuesdays, and tech news in The Pulse on Thursdays. The last 12 months of issues adds up to around 5-7 books’ worth of reading (about 550,000 words). Free subscribers also got well over a nonfiction book’s worth of reading in their inbox: the first parts of the Tuesday articles, and a full article monthly.
Today, we look back, and peer in to the future:
How The Pragmatic Engineer has evolved
Popular, interesting topics
What I learned about readers!
Notable events
Looking ahead, and The Pragmatic Engineer Podcast
Before we jump in: if you’re someone who enjoys podcasts, I have a treat for you. The Pragmatic Engineer Podcast is launching with the first episode next week. To get the first episode in your podcast player, subscribe on:
YouTube — you can also watch the podcast announcement here
… or your favorite player!
1. How The Pragmatic Engineer evolved in the last yearSoftware engineering is an ever-changing field where the best teams continuously experiment with new approaches, and change how they work. I’m also an engineer at heart, so it’s only natural to do the same with the newsletter.
Here’s what’s changed in recent months; some of which you might have noticed, and other things that happened behind the scenes.
Unshared details from interesting tech companiesSome companies attract media attention due to how well they execute, and what this publication tries to do is learn how exactly they achieve it. I aim to bring details straight from the source, talking exclusively with engineering teams at companies in the spotlight. The last 12 months has seen exclusive articles on:
OpenAI: How they scaled ChatGPT, and Inside OpenAI: how does ChatGPT ship so quickly?
Anthropic: How the company built Artifacts (and details on how it operates)
Stripe: Inside Stripe’s engineering culture
Meta: How the company built Threads with a small team in 6 months
Bluesky: How a tiny team built a Twitter/X competitor and Inside Bluesky’s engineering culture
Oxide: A hardware/software startup on “hard mode”
In terms of deep dives, it’s been the most prolific year yet for this publication. Doing lengthy articles which lift the lid on how leading teams get things done, can be challenging. Understandably, some companies do not normally share the details you’ve been able to read in this newsletter. At the heart of delivering articles like those above, is trust. Engineering leaders and software engineers ultimately feel they have scope to be candid with me, and that means a lot. Their trust in myself and The Pragmatic Engineer team is a very rewarding part of writing this newsletter.
More articles mixing deep research with pragmatic observationsSince February, it’s not just me doing the extensive research that goes into each article: Elin Nilsson is this publication’s first Tech Industry Researcher. Previously, Elin spent 7 years at Spotify, working in various mobile and platform engineering teams. Before, she interned at both Spotify as an Android developer and Google as a web/UX prototyper. She’s brought a whole lot of things onboard since she joined in February: like her infectious enthusiasm for nerding out about tech topics, diving deep into technologies and trends, and bringing order to the chaos of having too much information to process on any given topic.
Since she’s come onboard, there’s been more deeply researched pieces published than ever, and each one of them went into more depth. Some of the articles that started with Elin’s insightful research:
GenZ software engineers according to older colleagues, and what GenZ devs really think
Bluesky: How it was built, and Inside Bluesky’s engineering culture
How do AI software engineering agents really work?
AI tooling for software engineers: Reality check, Reality check (part 2) and Rolling LLM tools out company-wide (Part 3)
Thanks to Elin, we can take on ambitious projects that were simply not possible when the Pragmatic Engineer launched because the research can take months, We’re on a roll with Elin, so expect these to keep coming! You can always suggest interesting topics for us to potentially look into.
Goodbye “Scoop,” hello “Pulse”During the first two years of this newsletter, there was usually an article every Thursday called “The Scoop,” which rounded up interesting things in the industry, and occasionally broke big tech stories. But as of a year ago, I am no longer doing this and the name change to “The Pulse” was part of this shift. I shared details about the change:
‘The Pulse’ just better represents the mission of these articles, which is to help you keep an “ear to the ground” on what’s happening across Big Tech and at startups – sectors which regularly affect the whole tech industry.
I receive and validate plenty of interesting information from insiders at these companies, but my main focus is analyzing what’s going on in our industry; the implications of that and the opportunities for software engineers and tech businesses.
Names matter, and I feel “The Scoop” suggests a more gossipy, tabloid newspaper-style approach than what these articles actually deliver. Since The Scoop’s first issue I’ve focused on relevant industry changes and what they mean. I hope “The Pulse” makes the intent of this series clear: to provide an ear to the ground, and the latest analysis of the state of our industry.
This change has been one I’m very happy with; it’s also partly why we’ve been able to publish more company deep dives straight from tech companies themselves.
When writing “The Scoop”, tech companies and devs whom I met were never quite certain if I had my “journalist” hat on (and would leak what they shared as news), or my “analytical” software engineer’s hat.
This made people reluctant to share information that could potentially identify their company, or affect their career. This was taxing for me personally and I decided I don’t want to be in the “gossip” business, but in the software engineering one. For that reason, the change from The Scoop was made.
But while writing it, I learned a lot about journalism and met a lot of reporters. It’s a fascinating world, but one I don’t feel is for me. Also, the time I save on no longer verifying information for news articles, means there’s more time to dive deep into relevant, interesting software engineering topics.
Industry PulseIn the spirit of experimentation, I tried out a new section inside the Thursday “The Pulse” article. Industry Pulse is a roundup of tech events relevant for devs and EMs, with commentary.
It has delivered early insights on topics like The impact of Section 174 on the software industry, predicting pressure on commercial open source to make more money, the unfortunate “startup purge event,” analyzing why Amazon sunset several of its products in one fell swoop, and much more.
Readers feedback was very positive from the start. Also, I enjoy writing it and look forward to it every day: so it has stayed!
2. Popular, interesting topicsIn case you’re looking for articles to read or re-read, here’s a selection of some standout ones from the past 12 months, in my opinion.
Popular topicsThe end of 0% interest rates, and what this means for software engineers was the single most-read article of the past year. This article was in a highly-detailed 4-part series about what feels like the biggest industry change in years: the end of low interest rates.
It is rare for an economic change to have the kind of impact on an industry that higher interest rates are having. We’re finding out that tech startups and interest rates are more connected than many of us would like to think.
Measuring developer productivity? A response to McKinsey. Consultancy giant McKinsey is known for many things, but software engineering expertise is not one of them. Yet the globally known consultancy started selling advisory services about this: including the claim that they had developed a way to measure devs’ productivity.
Dev productivity is a notoriously tricky thing to accurately measure, so I teamed up with industry veteran for a grounded take on why measuring productivity is difficult, and sharing our suggestions on how to go about this kind of measurement.
Inside OpenAI: How does ChatGPT Ship So Quickly? Up to early 2024, OpenAI was dominating headlines by releasing new products and features with an incredible pace. The company was out-executing the likes of Google, Meta and every other AI startup. But how did they do it?
I sat down with Evan Morikawa, who headed up the Applied engineering team (the team also building ChatGPT). We learned how OpenAI operates ChatGPT like a “startup inside a startup,” engineering has an unusually tight integration with Research, has a high talent density in their San Francisco offices, and has a lot of smaller day-to-day habits that add up. While I advise to never blindly copy engineering practices: it’s helpful to understand what makes a startup like OpenAI execute so quickly.
Surprise uptick in software engineering recruitment. Recruitment activity tends to be quiet in the summer months. This is why I was surprised to hear from several developers that they are seeing a spike in LinkedIn reachouts in June and July – mostly from startups.
Talking with dozens of developers – and some hiring managers – we uncovered that startups seem to have increased their hiring pace for a variety of reasons. What was common is how they were all searching for experienced engineers, and were becoming more “bold” in their reachouts: pinging senior+ engineers working at Big Tech and well-funded startups, who are profiles that usually are more reluctant to move.
Other interesting topicsHere are articles that I especially enjoyed for one reason or the other. If you’ve not yet read them, they could be worth your time:
The past and future of modern backend practices. From the birth of the internet, through SOA and virtualization, to microservices, modular monoliths and beyond.
How Big Tech does Qualty Assurance (QA). Most Big Tech companies have no dedicated SDET, QA, or tester roles. How do they produce quality software? A look into how Microsoft, Google, Meta, Apple, Amazon, Uber and Netflix do it.
Stacked diffs (and why you should know about them). Meta and Google have been using stacking for closer to a decade: a coding workflow that is very efficient for small PRs. So what is stacking, and how come it’s not more widespread in the industry?
What is Secruity Engineering? A deep dive into the ever-changing field of security engineering; a domain that can feel intimidating to some software engineers.
Applied AI Software Engineering: RAG. Retrieval-Augmented Generation (RAG) is a common building block of AI software engineering. A deep dive into what it is, its limitations, and some alternative use cases.
Thriving as a Founding Engineer: Lessons from the Trenches. Being a founding engineer at an early-stage startup is a vastly different, broader role than many people think.
Adobe can’t buy Figma: the impact on the industry. Months after we covered Figma’s engineering culture, regulators blocked Adobe’s acquisition of Figma. This event was likely what prompted Big Tech to stop acquiring companies, and to do “talent raids” instead with AI companies they seek to purchase.
There’s more, of course! For the full list of published articles, browse past deepdives, or The Pulse issues.
3. What I learned about readers — about you!This year was the first time I asked details about who you are: in the form of a survey.
DisciplinesI write the newsletter for software engineers and engineering managers, and was curious to know if most readers are, indeed, in this field. Turns out this is broadly the case:
Readers, based on profession. Based on 2,395 responsesThe majority of readers are in the software engineering field – ranging from entry-level to senior/lead/staff and above roles – and a good portion is in engineering leadership/management (ranging from engineering managers to VP and C-level engineering leadership positions.)
I’m pleased to hear so many folks in product management following along – extrapolating 5% to the whole reader base would translate to closer to 37,500 such readers! I hope that topics on engineers’ mind like paying down tech debt or adopting engineering practices are helpful to understand even at the product level.
The “other” bucket still accounts for a lot of people – closer to 75,000 folks, looking at the number of readers! Commonly mentioned titles included data scientist/analyst, security engineer, and recruitment. More sparse mentions included marketing, sales – and retired folks!
Types of companiesThe publication covers Big Tech, startups and scaleups. Naturally, I was interested to see where readers work:
About 65% of readers indeed work at these types of companies. At the same time, there’s a healthy representation of people working at:
More traditional, but tech-heavy companies
Consultancies and developer agencies
More traditional, non-tech heavy companies
Bootstrapped companies (4.2%)
Academia and research labs (1.8%)
Public sector (1.5%)
Nonprofits (1.1%)
Other (~3%): self employed, founding a startup and in-between jobs were the most common mentions
InterestsI asked people for feedback on the publication, and gathered these observations:
Variation of topics makes the newsletter enjoyable. Many of you shared that you like it when topics are switched up. At the same time, the few times I did a multi-part series on a topic: I’ve received complaints from some of you how you’re ready to learn about other areas.
The “practical research” pieces are a definite hit. Lots of professionals shared how they appreciate getting what feels like a closer to real-time feedback on what is happening in the market, and the industry.
Disruptive technologies and companies: a common request. Many of you are asking to cover more about disruptions as we see them happen. Timely pieces like the ones on stacked diffs, modern developer productivity measurements, or companies like OpenAI are consistently popular, validating this ask.
4. Notable eventsA few other things this happened in year three, that are worth a mention:
The Software Engineer’s Guidebook releasedI started to write The Software Engineer’s Guidebook before starting the newsletter. In what is amusing irony: writing the newsletter delayed finishing the book – there’s only so much writing one can do, after all! Still, After four years of work, the book was finally released in print in November 2023, as an e-book in February 2024. The book became a #1 best seller across tech books on Amazon on launch. In the first 10 months, the book has seen more than 30,000 sales. Thank you to everyone who picked up a copy!
By the end of this year, I am aiming to release an audiobook version. Translations to 8 different languages are in progress (German, Japanese, Korean, Russian, Traditional Chinese, Simplified Chinese, Hungarian, Mongolian). The translations should be out between this fall and the spring of 2025 in respective countries.
I self published the book – as tech book publishers either passed on the title, or requested too many changes I did not agree with – and the self publishing process has been full of learnings. I’ll share more about this process, as self publishing, audiobook producing, and selling foreign rights has been full of learnings.
Leap DayThis year, February had 29 days. This small change caused a series of hiccups within software systems: from airlines to payment terminals. We shared a roundup of these. The next leap day will happen in 2028 – hopefully we’ll see more robust software running these critical systems by then.
The largest software-inflicted outageIn July of this year, we collectively witnessed the largest-ever software inflicted global outage where a routine Crowdstrike configuration update took airlines, banks and retailers offline across the globe.
We can now conclude that Crowdstrike did not do canarying or a staged rollout of this change, because they likely assumed that this change carries no risk. It’s always cheaper to learn from others’ mistakes: and this incident is a reminder that you should avoid doing “YOLO releases” when operating a critical line of business – even if that release is assumed to be safe to roll out.
5. Looking aheadSome of the best-received articles come from guest writers: software engineers with deep experience in a topic. They then share their stories, learnings, mental models in a way that is helpful for more of us. Guest posts are always paid, and I closely work with guest authors to create a polished and enjoyable final piece. I’m always on the lookout for new guest writers:
Express interest in writing a guest post
One last thing… The Pragmatic Engineer Podcast. In the first two years of The Pragmatic Engineer, I ran several interviews with engineering leaders. A few examples:
Platform teams and the platform team mindset with Ganesh Srinivasan
Platform teams and developer productivity with Adam Rogal, Director of Developer Platform at DoorDash
The full circle on developer productivity with Steve Yegge, formerly Head of Engineering at Sourcegraph
The last 12 months, there have been no similar interviews – even though they are interesting and useful. At the same time, I kept talking with interesting people in the software industry, and frequently wished I could share these conversations wider. This is how the idea of the podcast was born.
The Pragmatic Engineer Podcast is an addition to the newsletter. Every second week, on Wednesday, a new episode will be out.
This show covers software engineering and Big Tech and startups, from the inside. I do deepdives with experienced engineers and tech professionals who share their hard-earned lessons, interesting stories and advice they have on building software.
After each episode, you’ll walk away with pragmatic approaches you can use to build stuff – whether you are a software engineer, or a manager of engineers. Expect a similarly polished experience as what the newsletter already delivers. Please subscribe using your favorite podcast player, or sign up the podcast’s YouTube channel or on Spotify or on Apple Podcasts. The episodes will also be shared in the newsletter, under the “Podcast” category.
I hope you’ll enjoy this addition!
For the next year, expect more deeply researched and practical deepdives both for technologies and interesting tech companies. If you’d like to suggest a topic or a company to research, you can do it here:
Thank you – and see you in the next issue!
Welcome to The Pragmatic Engineer Podcast, hosted by Gergely Orosz, the author of The Pragmatic Engineer newsletter. In each episode, we dive deep into the world of software engineering, offering practical insights on scaling teams, engineering leadership, and navigating the evolving tech landscape. With industry veterans and successful engineers as guests, this podcast is perfect for anyone looking to level up their engineering career with real-world advice.
Subscribe to the podcast on YouTube, on Spotify, or Apple.
You can also subscribe to the newsletter here.
Welcome to the inaugural “Ask a [cybersecurity] futurist column. This month’s question comes from Anna Borg.
She asks:
How can we change the “rules in the game” and make it less interesting to use the internet for attacks on organisations and societies including democracy. We will never “win the game” if we are forced to play defence play 24/7, 365 days per year. How do we change the incentives for attackers?
I won’t sugar coat it, this is a complex and interconnected problem. Our increased cyber attacks are a symptom of a much larger problem. (Which means the solutions will not just address the cybersecurity problem at the core of this question, but bigger problems that society deals with.)
The Complexity of Cybersecurity ThreatsFor defense teams, the current state of cybersecurity feels like an endless battle, with organizations and societies constantly reacting to the latest threats. For hackers, it’s a landscape with abundant vulnerabilities if one is willing to put in the work to find (or create) the magic key — like a recent Advanced Persistent Teenager.
Cybersecurity is a long imbalanced game of whack-a-mole. Defenders have to guard everything, while attackers have the luxury of focusing. So how can you move out of a reactive mode to stay one step ahead of the next attack? And further, how can we make it even pointless for attackers to target us in the first place?
Understanding Hacker MotivationTo change the rules, we need to understand why cyber attacks are happening. What drives someone to become a black hat hacker? Or to start a black hat collective? Or join an elite nation state hacking team? I’ve spoken with security professionals, hackers and analyzed research data, and the motivations come down to: money, espionage, and the thrill of the con.
Viktor and Giselle from Better Call SaulIn a survey I conducted last year with identity-security experts, financial gain and espionage were the top motivations, followed by a desire for notoriety or the thrill of hacking. Most hackers today aren’t lone wolves; they’re part of organized collectives that pool their skills and resources to hit bigger, complex targets. They might even employ the Silicon Valley approved “AAS/as a service” business model.
There’s another factor that’s often overlooked: socioeconomic conditions. During my research for the paper Shifting Paradigms: Using Strategic Foresight to Plan for Security Evolution, I was genuinely surprised to hear about the overproduction of well-educated young people unable to find good jobs after their security education. There are very few well paid entry level security jobs — even in the US and developed nations.
Changing the IncentivesSo how do we change the incentives to make cyber-attacks less appealing and less profitable for would-be attackers?
I’m going to skip over the obvious answer of creating/using better security technology. Everyone is racing to implement better tech solutions, but this is just a band aid — admittedly a really big band aid. I’m going to talk about non-tech solutions.
Economic SolutionsIf it’s true that we are overproducing highly educated security people, could we solve some security problems by having full employment for all educated security experts?
One scenario in Shifting Paradigms envisioned this idea.
5.10 Scenario 9: Money, Cash, Hoes
In this scenario, whose name is inspired by the Jay Z song, security defensive work has become so attractive and well-paid that black-hat work is no longer attractive, and people with security-relevant technical skills are almost universally employed in defensive jobs. Governments can still hire offensive hackers, but criminal syndicates have a very hard time competing with white-hat employers.
Changes from Today: Employers pay great salaries, offer significant flexibility and benefits, and recruit extremely aggressively, especially in poor countries with good technical talent and in ungoverned spaces. There are many good entry level security jobs.
Could one of the most effective ways to reduce cybercrime to increase legitimate employment opportunities for those with technical skills? If well-paying jobs in cybersecurity or other tech fields were more readily available, especially in economically disadvantaged regions, would the incentive to engage in illegal activities diminish?
I think this scenario is illustrated in Wired’s recent article about Priscila Barbosa, Queen of the Rideshare Mafia, which describes the elaborate identity theft and fraud Priscila engaged in. Barbosa took IT classes back in her native Brazil and was a successful entrepreneur until the recession caused business to disappear. She came to find her fortune in the US — which she did. But because she overstayed her visa, she could not work legitimate jobs. I’d like to imagine, that if there was a legitimate way to work in the US, she would have never participated in criminal activities. And maybe, if there had been good tech jobs in Brazil, she might not have even needed to seek her fortune in the US.
In my view, Barbosa is a victim of economic lack of opportunity.
Disrupting [Criminal] Business ModelsWhat are the business models that make cybercrime profitable? I remember Kim Cameron, Identity Architect, saying (not sure if this is his quote or he was quoting someone), that “the most profitable business model on the internet is crime. Back in the 90s and again now.”
Even entrepreneurial criminals have embraced the successful “as a Service” software development model. But to me, this echoes back to the economic themes. If there were legitimate economic paths half as successful, would there be less interest in criminal activity?
Nation-State AttacksNation-state hackers are a whole different beast. Their motivations are tied to geopolitical loyalty and belief. To reduce these types of attacks, we need to focus on improving global cooperation and reducing international tensions. This might sound idealistic, but fostering peace and global coexistence could significantly decrease the incentive for nations to engage in cyber warfare.
Reducing the ThrillThen there’s the issue of hacking for fun, cultural subversion, or “the thrill of the con.” This is harder to combat, but one approach is to create more avenues for ethical hacking while embracing the dark side of hackers. Bug bounty programs, for example, allow hackers to flex their skills in a legal, constructive way. These programs are subversive in their own right but channel that energy into something positive.
Finding SolutionsIs the solution simply a transparent and interdependent global economy with legitimate well paid jobs so we can all live our best lives without stepping on someone to do so?
I don’t know about you, but that feels like a big and impossible solution — perhaps a bit too utopian. So, I want to leave Anna with some reasonable actions she or her company might take.
Entry level security roles: Increase well paid jobs & support development programs to transition people into well paid jobs. As part of this, eliminate the rockstar mentality, and hire someone who is good enough for the job and then treat them well. Embrace the Lulz factor: Embrace the shadow of cybersecurity by engaging and encouraging the exact hacker mindset but to improve security — and again, offer an economic incentive. Bug bounties are both subversive and helpful.Both of these ideas could have a corporate initiative with a budget, KPI and program manager. They could be funded out of HR or professional development, IT or risk management line items. Companies could partner with universities, hacker collectives and government agencies, maybe sponsoring a hackathon or participating in a local cyber wargame activity.
Got QuestionsIf you could see the future of cybersecurity, what would you ask? Submit your question for a chance to be featured in the next “Ask a Cybersecurity Futurist” column.
If you found this insightful please consider booking me for a keynote talk or a lunch and learn session. Or sign up for the Future of Cybersecurity newsletter.
Selective disclosure and user control: With digital identity, the relying party (a party requesting information, for example a car rental company or a merchant) is able to request only the relevant parts of a person’s ID. Today, if you’re presenting your physical ID (for example to confirm your age or your address) you have to share everything — your name, address, your physical description and more. However, with a digital ID, you can share only the required data. Additionally, you must authenticate the device with a fingerprint, PIN or passcode in order for any of your ID information to be shared with the requester.
選択的開示とユーザー制御:デジタル ID では、依拠当事者(情報を要求する当事者、たとえばレンタカー会社や商業者)は、個人の ID の関連部分のみを要求することができる。現在、物理的なIDを提示する場合(たとえば年齢や住所を確認する場合)、氏名、住所、身体的特徴など、すべてを共有しなければならない。しかし、デジタルIDでは、必要なデータのみを共有することができる。さらに、あなたのID情報を要求者と共有するためには、指紋、PIN、パスコードでデバイスを認証する必要があります。
選択的情報開示のUXがどうなるのか気になりますが、個人的な意見としてはリライングパーティが全体ではなく最低限の要求ができるようになるので、ウォレットとしてはそのリクエストに対応できるようにするよ、という話だけでは全然足りない気がしています。(実装する立場としては理解できますが)
あくまでユーザの意思によって開示する情報を選択できるという体験が重要だと思うので、リライングパーティがどんな属性を要求してきているかに関わらず、自身で開示する属性を選べる状態にはなっていないといけないと思います。(結果的にリライングパーティの要求を満たさなかったとしても)
また、オフラインでの利用についても考慮をしていってもらえるといいなぁ、、と思います。たとえば、画面を見せる際に検証者の目線では「これは正式な書類である」ということが視認できる状態が重要なので、表面は正式な証明書であることが視認できるだけ、タップして裏面を見せるとユーザがあらかじめ設定した開示したい最低限の情報だけが記載されている、という状態が作れるといいのではないかと思います。
いずれにしても3rdパーティウォレットを含むエコシステムが正常に出来上がるような規制などは政府が中心に整備してもらえるといいですね。ユーザーの声を正しく吸い上げるためにもAppleとGoogleだけに任せるのではなく、エコシステム全体として進化できていくことが重要な気がします。
Recently, I have been working on programming small modules for Retrieval Augmented Generation (RAG) using Open AI. Also recently, I did a Coursera class on advanced RAG and have taken in several videos and posts on it. I have used samples from those sources and tested using various LLMs to generate simple Python to perform RAG with Open AI. In general , I have been disappointed with the outcomes until I tried Azure Open AI with Azure AI semantic search.
DisappointmentMy general disappointments have come from the self-coded RAG attempts on a single PDF. The basic approach has been:
take a single PDF (not a small PDF — about 43,000 token when uploaded to Open AI API in whole and extract the text using PyPDF chunk the text using at best recursive text and/or sentence transformer; sometimes just naive simple character count split embed the chunks trying Chroma or text-embedding-3-small query the collection using Chroma or FAISS and in one instance also with a simple augmented prompt call the LLM with the initial prompt and context from the embeddings query the prompt was for a list of 10 principles in the document that were all outlined in a single paragraphIt’s nothing sophisticated by any stretch but they corresponded with the examples I had available.
The results — abysmal. I’m not really surprised. I’m not sure how such an approach could do very well with simple chunking, embeddings, and basically running off keyword proximity. But since these were supposed examples of the use of RAG I expected better results. There were only two times I received good results. One of those times I’ll outline below; the other was when I didn’t parse or embed and just passed the entire document as context. Of course, the latter worked well but that was not the exercise I was after.
SuccessMy successful attempt came when I didn’t create the code directly but used Microsoft Azure Open AI playground. I have no doubt that coding it up would have worked just as well since it relies on the Azure AI infrastructure and the code would be little more than passing a prompt to the Azure LLM instance and getting the results. Here is what it consisted of:
setup of an Azure Open AI instance using gpt-4o-mini model and, I think, the text-embedding-3-small embedding model setup an Azure AI Search instance with semantic search and indexing pointed to a blob storage container with a single PDF in itI think that was it. I then went to the Azure Open AI Studio playground grounded the model with the search instance, provided my simple prompt, and got back the desired results. Was it the semantic search that made it work well? I suspect it helped a lot. I need to try it without the semantic search and see what happens. Sorry, I forgot to try that scenario.
RecapAll in all, I was very disappointed with the RAG results, as I coded it, especially as they were based on examples or AI generated single document RAG code. But, I was very pleased with the Azure test and I think the semantic search made all the difference.
In the late ’70s, I worked for a while at the Psychical Research Foundation, which occupied a couple of houses on Duke University property and did scientific research into the possibility of life after death. My time there was a lever that has lifted my life on Earth ever since, including many deep and enduring friendships.
Bill Roll ran the place then, and many fascinating characters passed through the PRF orbit. Eda and Lawrence Lashan, for example. Mac McDonnell. John Fetzer. One of the most memorable was Iris, a brilliant woman from Savannah with a strong personality, dyed red hair, and lots of talents. Her surname was pronounced “Mock,” but spelled with an “a” or two. (I think it was Maack.) She also mentioned occasionally that her brother was Ken “Hawk” Harrelson, the baseball player I remember best from his peak years with the Boston Red Sox. For many years since then, however, he has been the play-by-play announcer for the Chicago White Sox. Now 83, Hawk is still calling games for what will almost certainly be the losing-ist team in major league history.
So this morning, after I read “Hawk Harrelson spent 3 decades calling the White Sox. Now he can’t stand to watch” in the NY Times (sorry, paywall), I looked up “Hawk Harrelson” plus “sister” and “Iris” and landed on this 2018 page, which has this passage from Hawk’s autobiography:
“My sister was a heavy smoker who had died of lung cancer when she was only 40. I was so proud of what Iris had done with her life after such a tough start, having to get married at the age of 14. She wrote a few books and became a gourmet cook. She also learned to speak fluent German, Russian, and Spanish. She lectured at Duke University and at the University of Toronto. She became an accomplished pianist. She also dabbled in acting and landed a few holes on stage in New York. And she was an interior designer, having turned my penthouse pad into a beautiful home. “But we had a falling out several years before she died when I turned down her request to borrow $250,000. She had wanted to open a nightclub in downtown Savannah and I didn’t have that much cash at the time. Plus, I didn’t think her business idea was a good one. When I refused her request, she walked out the door and I never saw her again. “Apparently, she never quit smoking.”While I’m not surprised to learn that Iris is gone (she was older than me, and I’m seventy-seven), it’s a shock to hear that she died so young, so long ago, and so full of talent and promise. I’m also not surprised that there is almost zero information about her on the Web, given that it was born after she died, and is a whiteboard as well. So maybe this post (titled with her birth name) will at least help make her more remarkable than I’ve made her so far.
This post shows how to use Elasticsearch in .NET Aspire. Elasticsearch is setup to use HTTPS with the dotnet developer certificates and and simple client can be implemented to query the data.
Code: https://github.com/damienbod/keycloak-backchannel
SetupTwo services are setup to run in .NET Aspire. The first service is the official Elasticsearch docker container and deployed using dotnet developer certificates. The second service is an ASP.NET Core application using the Elastic.Clients.Elasticsearch Nuget package. The App.Host project is used to set this up and to link the services together.
Elasticsearch development serverThe Elasticsearch container is configured in the program class of the App.Host project. The container is run using HTTPS and takes the Aspire parameters for configuration of the default account.
var elasticsearch = builder.AddElasticsearch("elasticsearch", password: passwordElastic) .WithDataVolume() .RunElasticWithHttpsDevCertificate(port: 9200);The developer certificates needs to be created and copied to the specific folder inside the Elasticsearch docker container. This is implemented using a shared folder and the Elasticsearch xpack.security.http.ssl properties are set to match. The following three properties are used:
xpack.security.http.ssl.enabled xpack.security.http.ssl.certificate xpack.security.http.ssl.key using System.Diagnostics; using System.IO.Hashing; using System.Text; namespace Aspire.Hosting; // original src: https://github.com/dotnet/aspire-samples/tree/damianedwards/keycloak-sample/samples/Keycloak public static class HostingElasticExtensions { public static IResourceBuilder<ElasticsearchResource> RunElasticWithHttpsDevCertificate(this IResourceBuilder<ElasticsearchResource> builder, int port = 9200, int targetPort = 9200) { if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { builder .RunElasticWithHttpsDevCertificate() .WithHttpsEndpoint(port: port, targetPort: targetPort) .WithEnvironment("QUARKUS_HTTP_HTTP2", "false"); } return builder; } public static IResourceBuilder<TResource> RunElasticWithHttpsDevCertificate<TResource>(this IResourceBuilder<TResource> builder) where TResource : IResourceWithEnvironment { const string DEV_CERT_DIR = "/usr/share/elasticsearch/config/certificates"; if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Export the ASP.NET Core HTTPS development certificate & private key to PEM files, bind mount them into the container // and configure it to use them via the specified environment variables. var (certPath, _) = ExportElasticDevCertificate(builder.ApplicationBuilder); var bindSource = Path.GetDirectoryName(certPath) ?? throw new UnreachableException(); if (builder.Resource is ContainerResource containerResource) { builder.ApplicationBuilder.CreateResourceBuilder(containerResource) .WithBindMount(bindSource, DEV_CERT_DIR, isReadOnly: false); } builder .WithEnvironment("xpack.security.http.ssl.enabled", "true") .WithEnvironment("xpack.security.http.ssl.certificate", $"{DEV_CERT_DIR}/dev-cert.pem") .WithEnvironment("xpack.security.http.ssl.key", $"{DEV_CERT_DIR}/dev-cert.key"); } return builder; } private static (string, string) ExportElasticDevCertificate(IDistributedApplicationBuilder builder) { var appNameHashBytes = XxHash64.Hash(Encoding.Unicode.GetBytes(builder.Environment.ApplicationName).AsSpan()); var appNameHash = BitConverter.ToString(appNameHashBytes).Replace("-", "").ToLowerInvariant(); var tempDir = Path.Combine(Path.GetTempPath(), $"aspire.{appNameHash}"); var certExportPath = Path.Combine(tempDir, "dev-cert.pem"); var certKeyExportPath = Path.Combine(tempDir, "dev-cert.key"); if (File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { // Certificate already exported, return the path. return (certExportPath, certKeyExportPath); } else if (Directory.Exists(tempDir)) { Directory.Delete(tempDir, recursive: true); } Directory.CreateDirectory(tempDir); var exportProcess = Process.Start("dotnet", $"dev-certs https --export-path \"{certExportPath}\" --format Pem --no-password"); var exited = exportProcess.WaitForExit(TimeSpan.FromSeconds(5)); if (exited && File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { return (certExportPath, certKeyExportPath); } else if (exportProcess.HasExited && exportProcess.ExitCode != 0) { throw new InvalidOperationException($"HTTPS dev certificate export failed with exit code {exportProcess.ExitCode}"); } else if (!exportProcess.HasExited) { exportProcess.Kill(true); throw new InvalidOperationException("HTTPS dev certificate export timed out"); } throw new InvalidOperationException("HTTPS dev certificate export failed for an unknown reason"); } }When the App.Host project is started, the Elasticsearch containers boot up and the server can be tested using the “_cat” HTTP Get requests or the default base URL will give a server information about Elasticsearch.
Elasticsearch clientThe Elasticsearch client was implemented using the Elastic.Clients.Elasticsearch Nuget package. The client project in .NET Aspire needs to reference the Elasticsearch server using the WithReference method.
builder.AddProject<Projects.ElasticsearchAuditTrail>( "elasticsearchaudittrail") .WithExternalHttpEndpoints() .WithReference(elasticsearch);Elasticsearch can be queried used a simple query search.
public async Task<IEnumerable<T>> QueryAuditLogs(string filter = "*", AuditTrailPaging auditTrailPaging = null) { var from = 0; var size = 10; EnsureElasticClient(_indexName, _options.Value); await EnsureAlias(); if (auditTrailPaging != null) { from = auditTrailPaging.Skip; size = auditTrailPaging.Size; if (size > 1000) { // max limit 1000 items size = 1000; } } var searchRequest = new SearchRequest<T>(Indices.Parse(_alias)) { Size = size, From = from, Query = new SimpleQueryStringQuery { Query = filter }, Sort = BuildSort() }; var searchResponse = await _elasticsearchClient .SearchAsync<T>(searchRequest); return searchResponse.Documents; }See the source code: https://github.com/damienbod/keycloak-backchannel/blob/main/AuditTrail/AuditTrailProvider.cs
NotesWith this setup, it is easy to develop using Elasticsearch as a container and no service needs to be implemented on the developer host PC. Setting up HTTPS is a little bit complicated and it would be nice to see this supported better. The development environment should be as close as possible to the deployed versions. HTTPS should be used in development.
Linkshttps://learn.microsoft.com/en-us/dotnet/aspire/search/elasticsearch-integration
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
https://www.elastic.co/products/elasticsearch
https://github.com/elastic/elasticsearch-net
こんにちは、富士榮です。
デジタル認証アプリと連携するサービス(事業者)一覧が大幅に更新されています。
https://services.digital.go.jp/auth-and-sign/case-studies/
2024年6月にアプリがリリースされた際は横浜市と三菱UFJ銀行のアプリの2つだけでしたが、この3ヶ月で15個まで増えています。
6月時点のポスト
https://idmlab.eidentity.jp/2024/06/blog-post.html
ざっとみていくと、
事業者や自治体のアプリそのものが連携するパターン 都市OSやIDaaSなどのプラットフォームが連携するパターン 本人確認アプリなど他の事業者アプリから呼び出されるアプリが連携するパターン に分類されそうです。マイナンバーカードを読み取るアプリがいっぱい https://idmlab.eidentity.jp/2024/08/blog-post_24.html デジタルクレデンシャルによる「本人確認」と「身元確認」 https://idmlab.eidentity.jp/2024/08/blog-post_14.html
Aaron Parecki and I published a new version the “OAuth 2.0 Protected Resource Metadata” specification that addresses the review comments received since the IETF Last Call. Per the history entries, the changes were:
Added metadata values declaring support for DPoP and mutual-TLS client certificate-bound access tokens. Added missing word caught during IANA review. Addressed ART, SecDir, and OpsDir review comments by Arnt Gulbrandsen, David Mandelberg, and Bo Wu, resulting in the following changes: Added step numbers to sequence diagram. Defined meaning of omittingbearer_methods_supported metadata
parameter.
Added internationalization of human-readable metadata values using the mechanism from [RFC7591].
Added resource_name
metadata parameter, paralleling client_name
in [RFC7591].
Added Security Considerations section on metadata caching.
Used and referenced Resource Identifier definition.
Added motivating example of an email client to intro.
The specification is available at:
https://www.ietf.org/archive/id/draft-ietf-oauth-resource-metadata-09.htmlSelf-Sovereign Identity (SSI) offers a robust and scale-able solution to the challenges of digital identity that is secure while also respecting privacy, freedom, and human rights. However, a mix of political and industry pressures, along with a push for quick adoption, risks turning SSI into just another marketing buzzword.
Nowhere is this trend more visible than in the mobile Drivers License (mDL) and the ongoing conversations around the second version of European Unions “electronic IDentification, Authentication and trust Services” (eIDAS).
Old Interests, New TechnologyThe first version of eIDAS was an adoption failure. While it was not providing privacy by design, technologically it was sufficiently correct to function. It did so by giving a central role to Trust Service Providers (TSPs) and Certificate Authorities (CAs).
These intermediaries sought to exploit their government issued monopoly and consequently choked off adoption for the first version of eIDAS.
In doing so, they repeated the same mistake made on the wider web, where excessive pricing led to stagnation in the adoption of encrypted communications. In 2014, only 10 short years ago, the vast majority of all connections was not using any kind of transport layer security (TLS).
HTTPS adoption became mainstream only between 2015 and 2019 — via Matthew Greene, article linked belowIt was the Snowden Revelations that changed everything. But not in the way the intermediaries had hoped.
While I am sure many were rubbing their hands, and expecting the pipe they were sitting on to finally start flowing, everyone in the industry knew that predatory pricing was to blame for the low adoption. So in November 2014, Let’s Encrypt was founded in order to provide for free what Certificate Authorities are selling at premium prices.
Today, Let’s Encrypt provides over 55% of all certificates used on the web. Which provides for two conclusions:
Let’s Encrypt has become the single point of failure for security on the internet. And the addressable market for TSPs has shrunk dramatically, and keeps shrinking.
Simultaneously, the costs for TSPs are growing. Not only in terms of technology cost. But also in terms of compliance cost. Because there is a whole industry of consultants and auditors that service TSPs, extracting six figure payments each year for initial certification and re-certification. Growing cost and shrinking markets are not a great prospect for any industry.
So when the new version of eIDAS came along, promising to switch to Self Sovereign Identity for privacy by design, security, and elimination of intermediaries, the TSPs deployed their lobbyists to Brussels to stop this threat to their remaining business.
The result was a bureaucratic monster, for which some people at DICE 2024 in Zurich voiced doubt whether it could even be implemented in practice.
Intermediaries are once again involved in every step. And all the technological choices are based on legacy Web2.0 technology shaped by the TSPs, without privacy by design, and with well known and regularly exploited security issues.
So TSP lobbying successfully defended their government mandated monopoly over the digital domain, and the fundamental issue that stopped adoption of the first version of eIDAS remains unresolved. Only the future will show whether this second attempt of the TSPs at monetizing all digital interactions will fare any better.
But that isn’t even the biggest problem.
The Web is a bad idea for personal identityThe web is a fundamentally unsound technology choice for personal identity, for a variety of reasons, starting with security.
The early web wasn’t really designed for privacy or security. Its original design was inspired by the need for institutional information sharing. Invented in 1989, security was added in 1995 when Netscape came up with Secure Sockets Layer (SSL), which led the way for today’s Transport Layer Security (TLS). This allowed encryption of communication, but all security was dependent on key management.
Because it was the only practical way at the time, intermediaries became the root of all trust for virtually all users. On top of those issues, secure communication requires two parties who may know nothing about one another to negotiate a secure key to use for encryption with one another. That negotiation must happen over the same channel that is being used for encrypted communication.
Which is a bit like yelling the combination of the key pad to your house through the open window over to your neighbour so he can come in and water the plants. If there are no eavesdroppers at that time: Great. If there are, they now also have the key. You’ll find out whether you were safe in a few weeks when you get back from vacation.
Diffie-Hellman key exchange was invented to solve that particular problem. But it is only secure if used correctly and keys are being rotated and refreshed often. Re-use of static keys or weak parameters can weaken security considerably.
On top of that, cryptography is an ever evolving field, and the need for web servers to support older browsers means that sometimes, outdated or known vulnerable settings are still being allowed. Managing all that is complex. It requires organisational level security teams.
And much like charging your modern Electric Vehicle using a coal powered steam engine to drive a generator, trying to fast track adoption for Self Sovereign Identity by using Web based Decentralized Identifier (DID) methods effectively negates all the conceptual and architectural progress that has been made with SSI.
All the underlying security issues our industry has struggled with for the past 30+ years, all the conceptual weaknesses of the Web, all the privacy issues and compromises also apply to SSI when using did:web and its descendants.
In practical terms this translates into most people once again finding themselves forced to entrust their personal identity to intermediaries in the form of custodial wallets and platforms. Just as they are doing today with Google, Facebook, Amazon, Apple, Microsoft.
Most people will rent usage of their own identities, but they will have no ownership or agency. And those intermediaries will once again have full visibility and control over everything people are doing online.
That is a steep price to pay. Is it worth it at least in terms of security?
Even these large organisations get it wrong often enough for data leaks and other issues to occur frequently. When using the Web for personal identity, these compromises will have consequences beyond anything we have seen so far. What happens when the compromise has enabled the attacker to provide legally valid signatures in your name to sign contracts, or consent to sharing your personal pictures with the wrong parties?
And that’s only the start of the valley of doom.
The Web is based on top of the Domain Name System (DNS). Which is a system in which you rent usage of certain human readable domains on a yearly basis. Without it, the Web falls apart.
So let’s assume enlightened users rent their own domains to become independent from the intermediaries. Now they can move between them in case one of them has issues, goes out of business, or gets acquired by a billionaire with a questionable political agenda. Just move your domain and you’re good, right?
Not so fast.
Not only is it possible you may lose your domain, either due to changes in DNS policy, legal aspects, such as trademarks, or because you’ve missed a yearly payment.
If a personal identity can be lost that easily, was it ever really yours?
But there is an even bigger issue.
The W3C Decentralized Identifier (DID) standard was designed to provide for unique, non-correlatable identifiers for each individual interaction. Which sounds great in theory. But how non-correlatable are did:tdw:johnsmith.eu:223499182 and did:tdw:johnsmith.eu:673403151?
Would you be able to tell they might both belong to the same person, whose name might be John Smith?
Because the domain is always an intrinsic part of each Web based DID method, none of them allow for personal ownership without complete loss of pseudonymity, and non-correlatability.
So whatever pseudonymity and privacy can be achieved leads back full circle to large intermediaries and custodial wallets — just like the federated identity infrastructure of today.
“Anakin Skywalker at a crossroads. Go down the Web based SSI method path, and become Darth Vader, or choose a world of privacy, freedom and security?” TL;DR: Don’t do it, Anakin!Web technologies are a fundamentally poor choice for personal identity.
They can work in domains where privacy and non-correlatability are not required, for instance organisational identity. But for personal identity, they compromise almost everything that Self Sovereign Identity (SSI) was intended and designed for.
You cannot retrofit privacy and security onto an architecture that wasn’t built for them. Just look at Email. It has the same problem. Decades of adding layers of complexity and auxiliary systems have kept moving the problem somewhere else in the stack, to another component. Lots of snake oil has been sold. Some people got very rich this way. But the fundamental issues remain unsolved.
We should not repeat that same mistake for Self Sovereign Identity.
No system that is intended for personal identity should ever use did:web, did:tdw, or any of the other web based DID specifications. The concepts of SSI can solve all the issues above. They can provide people with real agency and control over their own identity. But not when using the Web.
The solution is simple.
Stop using Web based DID methods for personal identity!Humans are not organizations. Stop using Web based DID methods for personal identity. did:web and its descendants are very useful for some organizational use cases, and for prototyping new ideas. They have NO place when it comes to personal identity, or use cases that require strong privacy or security.
There are (at least) three superior alternatives that come to mind, all of which can also provide a secondary channel for key negotiation.
Stay tuned.
米国時間2024年9月12日(日本時間9月13日)、グーグルが米国パスポートのグーグルウォレットへの搭載を発表しました。
近年、日常生活のあらゆる面でデジタル化が進んでいますが、その中でも特に注目を集めているのがデジタルID1です。Googleは昨年から一部の州発行のデジタルIDをGoogle Walletに保存できる機能の展開を開始しましたが、さらなる進化を遂げようとしています。
まもなく、Google Walletに新しいタイプのデジタルIDを導入するベータテストが始まります。この新機能により、より多くの人々が、より多くの場所でデジタルIDを作成し保存できるようになります。注目すべきは、今回から米国パスポートを使用してデジタルIDを作成できるようになることです。
このデジタルIDパスは、一部のTSA(運輸保安局)チェックポイントで使用可能となり、国内旅行時の空港での時間と手間を省くことができます。
デジタルIDパスの作成プロセスは非常に簡単です:
Google Walletアプリで「米国パスポートでIDパスを作成」というプロンプトを選択 パスポートの裏面にあるセキュリティチップをスキャンする指示に従う 本人確認のためにセルフィービデオを撮影 数分以内にIDパスの準備が整ったという通知を受け取る https://storage.googleapis.com/gweb-uniblog-publish-prod/original_videos/PassportinGoogleWallet_Revised_R3.mp4 Google Wallte へのパスポート登録の様子 (出所)Google現在、このIDパスは一部のTSAチェックポイントでのみ受け付けられていますが、Googleはパートナー企業と協力して、将来的にはアカウント復旧、本人確認、さらにはレンタカーの利用など、より多くの場面でデジタルIDを使用できるよう取り組んでいます。
ただし、この技術はまだ初期段階にあるため、Google WalletのデジタルIDは現時点では物理的なIDの代替とはなりません。旅行時には引き続き物理的なIDの携帯が必要です。
セキュリティ面では、Google Walletは「プライベート・バイ・デザイン」原則によって個人情報を保護するように設計されたAndroidの高度なセキュリティを使用してデジタルIDを安全に保管します。IDパスは暗号化されて保存され、利用には指紋認証やPIN、パスコードでの認証が必要となります。また、本人確認時には共有される情報を事前に確認できるため、ユーザーが情報を管理できます。
さらに、カリフォルニア州のAndroidユーザーは既に州発行のID2や運転免許証をGoogle Walletアプリに保存できるようになっており、今後数か月以内にアイオワ州、ニューメキシコ州、オハイオ州でも同様のサービスが展開される予定です。
コロラド州では、MyColoradoアプリ内に新しいリーダーを導入し、州内の企業がデジタルIDを安全かつ簡単に受け入れられるようにする取り組みも始まっています。
このように、GoogleはデジタルIDの利便性と適用範囲を拡大し続けており、私たちの日常生活をよりスマートで効率的なものに変えようとしています。デジタルIDの未来に、今後も注目が集まりそうです。
(参考資料)
New Google Wallet features for travelers and commuters (2024-09-12) How we’re working to make digital identity a reality for everyone (2024-09-12)Three decades ago was invited to participate in a collaborative effort to develop President Bill Clinton’s National Homeownership Strategy. Those 100 recommendations raised homeownership from…
The post Leverage $25K downpayment assistance to protect homebuyers & revitalize buyer agency movement first appeared on Real Estate Cafe.The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.
Today, we cover:
Industry pulse. Apple announces AI features you can’t use yet, Stripe is a hit with popular AI companies, Confluent buys an up-and-coming competitor before it grows into a threat, game studio…
運転免許証とマイナンバーカードの一体化が、まもなく現実のものとなります。 この画期的な変更は、私たちの日常生活に大きな影響を与える可能性を秘めています。
警察庁が9月12日に発表した新しい方針によると、2025年3月から「マイナ免許証」が導入される予定です。 これは、現在の運転免許証の機能をマイナンバーカードに統合するという画期的な取り組みです。
マイナ免許証制度の概要マイナンバーカードは、 2016年に導入された個人番号制度の一環として発行されている身分証明書です。 このカードには、すでに様々な個人情報が ICチップに記録されていますが、 新たに運転免許証の情報も追加されることになります。
具体的には、免許保有者の写真、免許の有効期間、取得している免許の種類、 そして眼鏡着用などの条件といった情報がマイナンバーカードのICチップに記録されます。 これにより、1枚のカードで身分証明と運転資格の証明が可能になります。
マイナンバーカードと運転免許証を一体化手続きは、運転免許センターや免許更新を扱う警察署で受け付ける方向で調整が進められています。これは、運転免許証の暗証番号を覚えている人は、スマホでできるようにして欲しいところではあります。
また、今の法律では免許取消や停止処分の対象者について、所在が分からず処分ができていない場合に警察官が対象者を見つけた場合、免許証を預かる制度があります。2025年3月24日以降はこの制度がなくなり、「出頭命令書」を出す制度に変わります。
マイナ免許証のメリットこの一体化には多くのメリットがあります。 特に注目すべきは、住所変更手続きの簡素化です。 現在、引っ越しの際には自治体と警察の両方に別々に届け出る必要がありますが、 マイナ免許証では自治体での手続きだけで完結します。
さらに、免許更新時の利便性も向上します。 優良運転者と一般運転者は、マイナポータルという個人向けサイトで講習動画を視聴するだけで、 更新時の講習を受けたとみなされます。 これにより、更新手続きにかかる時間と手間が大幅に削減されます。 ただし、すべての手続きがオンラインで完結するわけではありません。 写真撮影や視力検査などは、従来通り免許センターなどで実施されます。 これは、安全運転に必要な基本的な身体能力を確認する重要な過程だからです。
マイナ免許証の導入は任意であり、既存の運転免許証も併用して使用することができます。 しかし、マイナ免許証のみを保有する場合、 新規取得時や更新時の手数料が比較的安くなるという経済的なメリットもあります。
免許取得免許更新現行の免許証2350円2850円マイナ免許証1550円2100円現行+マイナ2450円2950円2025年3月24日以降の手数料 懸念点の解消この変更は、デジタル化が進む現代社会における行政サービスの効率化の一環と言えるでしょう。 すでに「マイナ保険証」として健康保険証の機能を統合する取り組みも進められており、 2024年7月末時点でマイナンバーカード保有者の80%が登録を完了しています。
誤まった紐付けについてしかし、このようなデジタル化の急速な進展には懸念の声も上がっています。 特に、個人情報の管理や情報セキュリティに関する不安は根強く存在します。 政府は、これらの不安に対して丁寧な説明と十分な対策を講じる必要があるでしょう。例えば、以前、マイナンバーカードと健康保険証の連携でミスが発生しました。マイナ免許証ではこのことの繰り返しを防ぐための対策が施されていることを訴えかけていく必要があるでしょう。現在検討されているのは、申請時に本人がマイナンバーカードと運転免許証の両方を持参し、係員が申請者の目の前で両方の顔写真が一致することを確認するということです。これでちゃんと脅威に対策できているのかは要検討ですが。
落とした・無くした時の再発行に時間がかかる点また、マイナ免許証に一本化した場合、これを落としたりした場合の再発行にかかる時間の問題も心配の一つです。車の運転には免許証の携帯が必要で、これに対応するために現行の免許証は落としたりした時に即日再発行が可能ですが、マイナンバーカードは1ヶ月半ほどかかることもあります。この間運転できなくなったら困ります。
これは、実は、現行の運転免許証を再発行してもらうことで回避できます。結局2枚に戻ってしまいますが。
今後の予定この新しいシステム(マイナ免許証)の運用開始に向けて、警察庁は道路交通法の改正を2025年3月24日に施行する予定です。さらに、2024年9月13日から30日間、この新しい運用方法について一般の人々から意見を募集します。
マイナ免許証の導入は、日本社会のデジタル化における重要なマイルストーンとなります。 この変革が私たちの生活にどのような影響を与えるのか、今後の展開に注目が集まります。
参考記事» 【マイナ免許証】来年3月24日開始へ マイナカードと運転免許証一体化 現行免許証の選択・併用も可能だが手数料はマイナ免許証より割高に?(FNNプライムオンライン(フジテレビ系)) – Yahoo!ニュース https://t.co/5lCDeGcip8
— Nat Sakimura/崎村夏彦 (@_nat) September 12, 2024
「マイナ免許証」25年3月導入へ 更新時講習オンラインに:日本経済新聞https://t.co/vImZKSbCp9
— Nat Sakimura/崎村夏彦 (@_nat) September 12, 2024
Ungarn hat den EU-Innenministern diese Woche einen neuen Vorschlag zur Einführung der Chatkontrolle vorgelegt, der von POLITICO geleakt wurde. Der Einsatz „künstlicher Intelligenz“ zur Suche nach verdächtigen Bildern und Chats soll Messenger- und Chatanbietern danach freigestellt bleiben. Zur Suche nach bekanntem illegalem Material bleibt es aber bei dem weltweit einzigartigen Vorhaben, die massenhafte, verdachtslose und nach BKA-Angaben höchst fehleranfällige Kontrolle aller privaten Chats für sämtliche Diensteanbieter verpflichtend zu machen, selbst wenn dazu die bisher sichere Ende-zu-Ende-Verschlüsselung von Messengerdiensten ausgehebelt werden muss. Die EU-Regierungen sollen sich bis zum 23. September zu dem Vorstoß positionieren, die EU-Innenminister sollen ihn am 10. Oktober beschließen. Die Messengeranbieter Signal und Threema haben bereits angekündigt, solche Überwachungsfunktionen keinesfalls in ihre Apps einzubauen und ihre Dienste in der EU gegebenenfalls komplett einzustellen.
“Statt Kindern durch sicherere Apps zu helfen sich im Netz vor Erpressung und Ausbeutung zu schützen, werden Missbrauchsopfer durch einen realitätsfremden und vor Gericht zum Scheitern verurteilten Gesetzentwurf im Regen stehen gelassen,” kritisiert Dr. Patrick Breyer, ehemaliger Europaabgeordneter der Piratenpartei und Mitverhandler der kritischen Position des Europäischen Parlaments. “Unsere Polizei mit großteils irrelevanten Hinweisen auf längst bekanntes Material zu überfluten, stoppt keinen Missbrauch, sondern erschwert seine Aufdeckung zusätzlich. Falls diese Chatkontrolle kommt, werden wir Bürger gängige sichere Messenger nicht mehr nutzen können – das bedeutet wir verlieren den Kontakt zu unseren Freunden und Kollegen in der ganzen Welt. Wollen wir wirklich, dass die EU weltweit führend beim Abhören unserer Smartphones und der flächendeckenden Überwachung der Chats von Millionen gesetzestreuer Bürger wird?”
Breyer ruft dazu auf, Druck auf Bundesinnenministerin Faeser und Bundesjustizminister Buschmann zu machen: “Nur dagegen zu sein und dann überstimmt zu werden, reicht nicht zur Verteidigung unseres digitalen Briefgeheimnisses und sicherer Verschlüsselung. Die Bundesregierung muss jetzt dringend Verbündete suchen, um eine Sperrminorität und eine grundlegende Überarbeitung zu erreichen. Die Sicherheit unserer Kommunikation steht auf dem Spiel!”
Info-Website: https://www.chatkontrolle.de
Hilf jetzt mit die Chatkontrolle zu stoppenThirteenth in the News Commons series.
I grabbed the spottedhawk.org domain after hearing Garrison Keilor read this passage from Walt Whitman’s Song of Myself over Leo Kottke improvising on guitar:
The spotted hawk swoops by and accuses me.
He complains of my gab and my loitering.
I too am not a bit tamed. I too am untranslatable.
I sound my barbaric yawp over the roofs of the world.
Most of what I do is in deficient obedience to Whitman’s spotted hawk. Including this blog.
Early in this millennium, when it was as easy to blog as it later was to tweet, I blogged constantly. The stretch from 1999 to 2007 was blogging’s golden age, though we didn’t know it at the time. (My blog from that time is archived at weblog.searls.com.) My blog then was a daily journal, and in a literal way that made me even more of the journalist I had always been.
On that career side, I was also employed for all that time by Linux Journal. My name was on its masthead for twenty-four years, from 1996 to 2019. When LJ was sold at the end of that stretch, I left as editor-in-chief. After that, I was the host of FLOSS Weekly on the TWiT network. Both were paid gigs, and when the FLOSS Weekly gig ended last December, so did my long career in journalism.
And maybe that happened just in time, because journalism has since then acquired a taint. In this past weekend’s Cornerstone of Democracy newsletter, Dan Gillmor sources Josh Marshall of Talking Points Memo on the topic:
I guess I would say that as journalists our core mission is fundamental honesty with readers. That means always telling readers the truth, an accurate story as nearly as we are able to uncover it and understand it, as well as being transparent with the values, commitments and core beliefs we bring to the work we do. We believe in always being fair to everyone and everything we write about. Fairness is really only another permutation of accuracy. Balance is a construct applied after the fact that is often as not at odds with accuracy. A belief in democratic republicanism or civic democracy has always been at the core of what we do. It’s central to what stories we choose to focus on, it’s a value structure that permeates our organizational approach to what we do. I can’t speak for everyone at TPM. But as founder and still a guiding light, I think our understanding of what journalism is or should be is inextricably connected with democratic republicanism/civic democracy. I don’t think I would say we’re activists for democracy. But to me being on the side of civic democracy is inextricably connected to what we do and who we are. We’re on the side of civic democracy as much as we’re on the side of journalism.
I don’t want to label other journalists. But to the extent many other journalists don’t operate in this way, or understand their job this way, it’s because they work for publications whose business models simply aren’t compatible with this approach to journalism. What we now commonly call “both-sidesism” is rooted in the business structure of most contemporary journalism, specifically the need to have purchase across of wide ideological spectrum of the population.This is especially so in large corporate journalism because the need for scale and advertiser buy-in really requires that access to all parts or most parts of the political spectrum. In a hyperpolarized political culture when one side veers off the democratic rail that inevitably pulls a lot of that kind of journalism with it. I see that as fundamentally a structural reality. This goes back many decades, long before the current moment. It was more workable in a different political climate. But in this one that’s how it works. Journalists think they’re following core journalistic principles. But it’s driven really by business models.
That’s why now I just call myself a writer. Or a photographer.† Unless it’s to write a book (which typically pays little or nothing), to be a paid journalist today requires frequent or constant production of “content.” But I’d rather yawp, like I am now, here, or over at ProjectVRM, TrunkLine (a blog on infrastructure), or Customer Commons (which will in the next year become more important than everything else I’m doing).
All three of those involve work, but don’t pay. Same goes for serving as Chief Intention Officer of Kwaai.ai, a commuity of 600+ volunteers working on personal AI. (Check it out. Good stuff going on there.)
But before I give up on the journalist label, let’s ask Perplexity.ai* to answer What is the purpose of journalism? Here’s the answer I just got:
Based on the search results, the primary purpose of journalism can be summarized as follows:
To provide citizens with accurate and reliable information they need to function in a free society. This is considered the central purpose of journalism. To inform the public about current events, issues, and developments happening in their communities and around the world. To serve as an independent monitor of power and act as a watchdog over those in positions of authority. To provide a forum for public criticism, debate, and compromise on important issues. To convey context and help citizens understand the significance of events. To set the tone for public discourse and help define community goals and values. To empower citizens to make informed decisions about their lives and society. To facilitate democratic processes by educating voters and inspiring civic engagement. To support commerce by creating awareness about products and services. To drive cultural transformation by connecting people to communities and ideas beyond their immediate surroundings.Journalism is considered essential because it helps citizens navigate their lives, make important decisions, and participate effectively in democratic societies. It serves as a crucial source of verified information in an increasingly complex world, helping to separate fact from fiction and provide context to current events.
Is that what journalists are doing today? Was it ever? Really? All of it?
I don’t know. Mostly journalists write stories. If you’ve read Where Journalism Fails, or Stories vs. Facts, you know I think the story-telling imperative has its own problems.
Back in the early ’90s, when I was doing PR for a hot Silicon Valley startup, I asked a top reporter at one of the tech magazines why he didn’t want to see my client’s brilliant and charismatic marketing director. The reporter said, “He abuses the principle of instrumentality.” When I pressed the reporter to expand on that, he explained that everyone involved knows that reporters are used as instruments by whoever spins them. The “principle of instrumentality” is about knowing, and trying to ignore, the simple fact that journalism is instrumented in many ways. While Josh Marshall talks above about the instrumenting of journalism by business models, in this reporter’s case, it was by the persuasive charisma of a strong personality who wanted positive coverage.
I realized then that I wasn’t being hired at the same magazine (or at any publication before Linux Journal, and I pitched many) because I didn’t want to be an instrument. More specifically, they all wanted me to write sports stories about competing companies. “Apple vs. Microsoft,” for example. “Motorola vs. Intel.” I wanted to write about interesting stuff without favor to anybody or anything other than what seemed right, important, fun, or just interesting. Sports coverage of tech didn’t do it for me. Nor was any of the other usual stuff. Linux was a cause, however, so I worked to make my Linux Journal writing as non-evangelical as possible, though I did get credited with helping put both Linux and open source on the map.
Was I a journalist while working as an editor there? I suppose so, given that my work hit at least some of the ten items above. At least I thought of myself that way.
A difference today is that we are all both digital and physical beings. Here in the digital world (where I am now), anybody can publish anything, on many different platforms, including their own if they’re geeky enough to make that work. According to the Podcast Index, there are 4,262,711 podcasts right now. Instagram has over two billion users. Says here there are over three billion blog posts published every year, and over six hundred million active bloggers. (I suppose I am three of them.) The same piece says “Over 90% of blog posts receive zero traffic.” Many of those blogs are faked-up, of course, but it’s still clear that the world of online publishing is a red ocean, while Mastodon, Threads, Bluesky, Nostr, and the like are more like small rivers or bays than one blue ocean. (Links in that last sentence go to my tiny presence in each. I’m also still on Xitter and Linkedin, for what those are worth.)
So now I’m thinking about what principles, old and new, work in the digital media environment and not so much in the old analog one.
Here’s one: We’re past the era of “What’s on.”** Unless it’s a live sport or some other kind of scheduled must-see or must-hear event, you can catch it later, on your own time.
Here’s another: We don’t have to fill time and space with a sum of “content.” We don’t. Yes, it helps to have a schedule and be predictable. But it’s not necessary, especially if you’re being paid little or nothing.
Here’s another: The challenge now is building and maintaining an archive of facts, and not just of stories. I’ve written about this elsewhere in this series. Go look it up.
Another: Try to grab as many of those facts as you can before and after they turn into stories or don’t. This is what calendars are for. Even if nothing comes out of a meeting or an event that appears on a calendar, it’s good to know that something happened. And to archive that as well.
I also believe both of those principles are easiest to apply in local contexts, simply because there is a relatively finite sum of facts to work with locally, and facts still matter there. (Scott Adams tells us they don’t in the wider world. And he has a case.)
This is one reason I’m embedded in Bloomington, Indiana. We’re working on all that stuff here.
† My photos here and here on Flickr have about twenty millon views so far. The last peak was five thousand on Sunday. The top attraction that day was this shot of Chicago I got on a cross-country flight between Phoenix and Boston in 2011. That one photo has logged 26,159 views so far. All my photographs are CC-licensed and free to use, which is why over 4,000 of them are in Wikimedia Commons, a library of images used in Wikipedia. So thousands of those (it’s hard to tell exactly how many) end up in Wikipedia. Many more accompany news stories, such as this one from Lawrence, Mass, this one from a power plant, this one from a lithium mine, and all these from Chicago. And I put none of them in either Wikimedia Commons or Wikipedia. Other people do that. I just put the photos out there. Meanwhile, this blog maxed at a little over 300 views one day last week, but usually gets a dozen or so. My old blog ran at about five thousand a day, and sometimes ten times that many. To bring this back to the theme of this post, while I do a lot of photography, I don’t think of myself as a photographer. I take pictures. And I write. And I talk some. All are for the same purpose: to be useful and to make stuff happen. Good stuff, hopefully.
By the way, the bird at the top is a juvenile red-tailed hawk. There is no one species called a spotted hawk, so this one will have to do. I shot this bird a couple of months ago, perched on a 480-volt line in the alley behind our house in Bloomington, Indiana. I was looking to hear a barbaric yawp, but he, or she, failed me on that one.
*I first asked ChatGPT 4o and got a cliché’d answer with no sources. Perplexity gave me a longer answer, just as cliché’d, but with eight sources. For fun, go ask both and see what you get. Try Claude and Gemini too. No two will be the same. Some will be better, some worse.
**I’ve written about this, but haven’t published it yet. Stay tuned.
Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:
Before we start: you can now see use “table of contents” quick navigation on the right side of each article, when reading the newsletter on the web. Just click on the sidebar, and you can navigate this article — and all other The Pragmatic Engineer articles. See it in action on the web. Happy browsing!
There’s no shortage of big claims about what LLM tools will be able to do, or should be able to do in the software engineering field. But what do they actually do, right now? We asked software engineers who regularly use these tools, and engineering leaders who oversee these tools in their organizations.
This article is based on a survey of 216 professionals and is the third and final part of a mini-series on GenAI tooling. It covers how these tools are being used ‘day-to-day’ in tech workplaces, and what engineers think about them. Today, we cover:
AI usage guidelines. A quarter of respondents follow company-wide usage guidelines. Interestingly, a minority of companies have banned GenAI tools over security and copyright worries.
Internal LLMs at Meta, Microsoft, Netflix, Pinterest, Stripe. Large, fast-moving companies not only embrace GenAI tools, but build their own internal versions for their engineers. Vendors are starting to offer similar boxed products.
Reservations and concerns. Most common reservations, and how to overcome them – and why devs tend to start using LLMs regularly.
Advice for devs to get started with AI tools. Start small, verify outputs, don’t “outsource” coding and other advice.
Advice for engineering leaders to roll out AI tooling, org-wide. A roundup of how companies adopted these tools successfully: guidelines, tooling, training, and how these impact junior engineers.
Measuring the impact of GenAI tools. Most engineering leaders say these tools have no visible or measurable impact – at least not yet. We suspect this is because the impact is hard to measure: and not due to the lack of impact.
AI strategy. Why do companies incorporate GenAI into their software engineering workflow? Experimentation and hoping for increased productivity are two big reasons.
In Part 1 of this series, we covered:
Survey overview
Popular software engineering AI tools
AI-assisted software engineering workflows
The good
The bad
What’s changed since last year?
Part 2 was about:
What are AI tools similar to?
State of AI tooling in 2024: opinions
Critiques of AI tools
Changing views about AI tooling over time
Which tasks can AI already replace?
Time saved – and what it’s used for
Now, let’s dive into this final part of this mini-series.
1. AI usage guidelines across companiesWe asked survey participants “how is AI tooling used for development at your company?” The responses reveal different approaches:
Common ways LLM tooling is used at tech companiesThe most referenced approaches:
No formal guidelines. Around 25% of respondents (53 out of 216) say their company has no recommendations about AI tooling. People use it as they see fit.
Some guidelines. 25% (50 respondents) say their workplaces have rules and guidelines for AI tooling.
AI tools banned. Circa 12.5% (25 responses) say their businesses ban usage of AI tools, mostly due to concerns about code security, and potential copyright infringement. We previously covered how several open source projects have banned AI-generated code commits for this reason.
Working on guidelines. 7% of respondents (15 people) share that their company is trialing AI tooling, or is in the process of adopting guidelines.
Strongly encourage AI tool usage. 6% of respondents (12 people) work at places which encourage using these tools wherever possible.
Common features of guidelines across workplaces, based on survey responses:
Specifying which tools and LLM models may be used
No inputting of sensitive information into AI tools like ChatGTP
No entering of internal (closed-sourced) code into AI chat tools
It’s pretty clear some guidelines are responses to fears that LLMs may retain the data employees input and use it for training. This is also a reason why a handful of respondents shared that their companies go through the added complexity of running LLMs on their own infrastructure. It’s a reminder that LLM solutions which don’t store company data have a strong selling point for tech companies.
2. Internal LLMs at Meta, Netflix, Pinterest, StripeOnly a fraction of respondents say their companies strongly encourage the use of LLM tools, but some of these are cutting-edge market leaders in tech. Let’s take a look at how a well-built internal LLM can help a business.
MetaThe social media giant has been investing heavily in ML and AI since before ChatGPT was released. Back in 2022, we covered how Meta was already preparing for AI/ML ‘wartime’ by investing heavily both in AI hardware, and hiring large numbers of AI and ML engineers. This investment has not slowed down since, and it’s little surprise that Meta seems to have built one of the leading in-house AI tools.
Meta’s internal tool is called Metamate. Director of Engineering Esther Crawford describes it:
“It’s an AI for employees that’s trained on an enormous corpus of internal company docs. I use it all the time for efficiency gains.
Any sizable company operating without an internal AI tool is already behind the curve.”
Esther explains what Metamate does:
“It has a ton of capabilities from summarizing to coding. Simple use cases:
Summarizing docs
Recapping work
Finding info across wikis
Exploring metrics and visualizing data
PSC feedback editor
Ask about diffs, SEVs, Deltoids, Tasks
Do the math for things like calculating capacity
Getting project status updates from GSD
Create queries, and so much more.”
Here’s a practical example on how useful Meta’s tool is, from Shana Britt E, director of strategic initiatives:
Microsoft“Recent use case: Performance reviews. Writing self-review, cleaning up peer reviews. For self-review, it can capture your diffs landed, status updates about your work from documents you published, etc. and puts it in a nice summary that you can then review and edit.”
The company offers Microsoft Copilot for Microsoft 365 for enterprises, and is dogfooding this system inside the company. I talked with software engineers who confirmed that the internal Microsoft Copilot is integrated with internal documents, and can thus provide more relevant context. It is also used in places like pull request reviews – although for this use case, I heard it’s more of a hit-and-miss in the quality of feedback.
StripeThe fintech company has a similar system to Metamate. Miles Matthias, product manager, shares:
Netflix“We have something similar [to Metamate] at Stripe and I spend a bunch of my time talking to it. I can imagine a world where I’m basically having a voice conversation with it all day every day as ‘work’ - especially when agents boom.”
The company has a place to access Netflix-provided versions of LLMs. A senior software engineer told us:
“There are AI guidelines, and corporate-provided versions of GPT, Claude and other models in a unified interface. People can share prompts that they find useful to colleagues.
My org is also exploring AI for our specific use cases, but thus far have not found any AI tools to be where we need. There is an opportunity to automate some manual business processes and we thought GenAI could help, but it seems traditional engineered solutions are still much better than GenAI."
The company builds internal LLM tools. One clever utility is called Text-to-SQL: a feature where internal users can use plain text to ask for a type of query, and the tool generates the right SQL to be used with the company’s internal data store called Querybook. The engineering team improved the first version with RAG, to help identify the right table names to use (we previously did a deepdive on applied RAG). The results are promising. As the company shares:
Vendors offering similar capabilities“We find a 35% improvement in task completion speed for writing SQL queries using AI assistance.”
There are plenty of vendors offering a “Metamate-like” experience out of the box. Glean seems to be the leader in this area. Other options include Village Labs, Microsoft Copilot for M365, Coveo and Akooda. This category is relatively new and there are plenty of up-and-coming startups. Search for terms like “AI company knowledge management tools” to find them.
The productivity perception of these systems rarely matches reality. Despite being a leader in the AI field, Meta is just figuring out how these tools can help it operate more efficiently. Metamate sounds impressive – and it’s ahead of what most companies have – but it doesn’t work optimally just yet, as we hear. I got this detail from talking with current engineers working at Meta.
The reason companies like Meta are investing so much into this area was articulated by CEO Mark Zuckerberg two months ago, on the company’s earnings call. He talked about how AI investments will take years to pay off, and Meta wants to be early. He said:
3. Reservations and concerns“You can also get a pretty good sense of when things are going to work years in advance. And I think that the people who bet on those early indicators tend to do pretty well, which is why I wanted to share in my comments the early indicator that we had on Meta AI, which is [...] early.”
When starting to use AI tooling, companies and developers often need to overcome reservations, or find workarounds. Let’s start by summarizing these reservations.
Reasons for not using AI toolingReasons for disallowing – or heavily limiting – AI tools include security and privacy worries; especially about internal, confidential information, and proprietary code being leaked. A few respondents also mention customer data.
Several larger companies have worked around these concerns by using in-house, self-hosted, LLMs, and their security and compliance teams add filtering to the inputs and outputs of these tools. This approach is clever:
Security and compliance teams can tweak filters to catch confidential or customer information that shouldn’t be shared
If confidential information is fed into a self-hosted model, this data does not leave the company to an external vendor
The obvious downside is that it’s not trivial to build and maintain. However, given that leading tech companies already have internal models and are heavy users, it’s likely other businesses will follow by either building in house, or using a vendor offering hosted LLMs with capability for internal security teams to tweak filters.
Developers’ reservationsBut it’s not just companies dragging their feet; developers are also hesitant about LLMs in the survey:
Commonly cited ethical and environmental concerns:
“The model used to power most AIs represents a large theft of labor from the commons, all to deliver increasingly terrible results.” – Senior software engineer, 19 YOE
“I have ethical concerns about code theft, and environmental concerns about energy consumption.” – Project lead, 9 YOE
“I feel their massive energy use goes against my personal code of ethics” – Software engineer, 8 YOE
“I am uncomfortable with its resource and energy usage, biases and AI hype, as ways to concentrate even more money and power at big tech companies and their culty leaders, which feels too adjacent to the Blockchain hype and grifts from a few years back for comfort.” – Software engineer, 40 YOE
These criticisms are valid. Large language models are known to be trained on copyrighted code, as well as on copyleft-licensed code, where the license is not complied with. And the surge in energy usage is also real, as covered in Is GenAI creating more carbon pollution by cloud providers?:
“It appears that the latest technological wave of GenAI may be getting in the way of corporate climate goals. Large language models (LLM) are very hardware and energy-intensive, and Azure, Google Cloud and AWS have all rapidly expanded their data center capacity and power usage, in order to meet demand. With energy usage surging, so are carbon emissions; which is the opposite direction from what they need to be going in, if these companies are to hit Net Zero in 2030 or any time after.
Google: carbon emissions up 48% in 2023, compared to 2019
Microsoft: carbon emissions up 30% in 2023, compared to 2020.”
There are clear benefits to GenAI, but also technological downsides. The ethical concerns seem to have no easy answers, while the history of computing has been about making computers more energy efficient, so we should expect the same here. At the same time, it’s concerning that GenAI is used to justify creating data centers which consume massive amounts of energy, or considering nuclear-powered data centers to keep up with computing demand.
Not enough utility, yet: We previously summarized negative sentiments in “Unimpressed” critiques in Part 2 of this survey. Common complaints about AI from engineers include:
Useful for simple stuff only, poor performance in more complex tasks
Little use outside of repetitive changes and boilerplate generation
Unreliable due to generating buggy code
Seen as a “fancy autocomplete”
More a hindrance than a help
Tedious to work with
Here are two more comments from engineers who stopped using AI tools:
“Seems useful for certain tasks, particularly writing related. For specific coding I could see it being used to generate more boilerplate, but personally my team tends to invest more in common libraries that reduce boilerplate anyway (while ensuring best practices are followed)” – Senior software engineer, 5 YOE
“ChatGPT is a novel tool with some potential to speed up boilerplate work and learning/investigation. It is not a high value for expert software engineers yet, but I’m optimistic that it will improve in a few years.” – Principal software engineer 20 YOE
These reservations are valid, but survey responses show that using LLM tools for 6+ months changes the views of many developers: mostly to a more positive, or more grounded, viewpoint. If you have an underwhelming first impression of these tools, it might be worth trying them daily for a bit before making up your mind.
Why do devs start using LLMs?We asked tech professionals why they started using these tools. The most common responses listed by frequency:
Company pushes LLM usage. Several large businesses set targets for departments of numbers of developers using LLM tools. Companies buying GitHub Copilot licenses also pushed workers to onboard. We’ve heard about company mandates, LLM training, and management expecting devs to use these tools for productivity.
To be more efficient / lazy to google it. Developers may adopt these tools to become more efficient, or because they can’t be bothered doing a web search. All found the tools help them get unblocked faster.
Peer pressure. Friends and peers in tech, and cofounders at startups, recommended them
Hype. Non-stop news about AI played a role in influencing software engineers to check out how the new technology works.
Pressure to keep up. Not wanting to fall behind in the industry, while seeing others use AI tools.
Curiosity. Discovering how the tech can help with their work, or how much better (or worse) it works compared to their existing workflow (especially versus Googling when solving problems, or using Stack Overflow)
An interesting detail for us is that company mandates and pushes are the single most-cited reasons for starting to use AI tools. It seems these do work – at least for that initial “push” to give the tools a go.
4. Advice for devs to get started with AI tools2024年9月10日、戸籍にフリガナを記載する改正戸籍法1の施行日が2025年5月26日と閣議決定されました。これにより、戸籍へのフリガナ記載に向けた準備が本格化します。
改正の背景と目的この法改正の主な目的は、
行政のデジタル化基盤整備の促進 本人確認情報としての利用 各種規制の潜脱行為の防止の3点です。
現在の戸籍では氏名の読み方が不明確な場合があり、行政手続きや本人確認の場面で不都合が生じていました。
新制度の概要2025年5月26日以降、戸籍にカタカナでフリガナが記載されるようになります。主な変更点は以下の通りです:
新生児の場合、出生届に記載されたフリガナが戸籍に反映されます。 すでに戸籍に氏名が記載されている人も、フリガナの届け出が必要になります。 フリガナには一定の基準が設けられます。 届け出の方法と期限 制度開始から1年以内に「振り仮名の届出」をする必要があります。 戸籍の筆頭者は氏名、それ以外の人は名前のフリガナを届け出られます。 届け出は書面またはマイナポータル(予定)を通じて行うことができます。 注意点 届け出がない場合、市区町村長により職権でフリガナが記載される可能性があります。現在使っている読み方と異なる読み方が登録されてしまう可能性があります。 フリガナは一般に認められる読み方でなければなりません。つまり、キラキラネームはNGです。逆に、一般の読み方以外の読み方を現在利用しておられる方は、パスポートや預金通帳など証拠を提示することによってそれを登録することができるようになります。 経過措置として、一度のみ家庭裁判所の許可なしでフリガナの変更が可能です。 自治体の対応施行に向けて、自治体は以下の準備を進めています:
正確かつ迅速な通知発送:本籍地の市町村長が戸籍に氏名の振り仮名を記載する前提として、戸籍に記載される予定の氏名の振り仮名等を認識する機会を確保することとしています。具体的には、住民票において市町村が事務処理の用に供するため便宜上保有する情報等を参考に、本籍地の市町村長から皆様に、氏名の振り仮名に関する情報を通知することとしています。この通知は、改正法の施行日(令和7年5月26日)から遅滞なく送付することとしています。 オンラインとアナログの併用:氏名の振り仮名の届出については、当該届出をする者の本籍地又は所在地の市町村に行うこととなりますが、窓口への出頭及び郵送による方法が可能です。また、マイナポータルを利用する方法についても現在検討中です。この制度変更により、戸籍におけるフリガナの取り扱いが明確になり、行政手続きの効率化や本人確認の精度向上が期待されています。市民の皆様は、制度開始後1年以内にフリガナの届け出を行うことを忘れずに、新しい制度に対応していく必要があります。
詳しい情報は、法務省のサイトをご覧ください。
【9月11日追記】総務省から住民記録システム標準仕様書【第5.1版】等の公表がありました
令和6年9月11日
住民記録システム標準仕様書【第5.1版】等の公表「自治体システム等標準化検討会(住民記録システム等標準化検討会)」(座長:庄司昌彦 武蔵大学社会学部教授)において、住民記録システム標準仕様書【第5.1版】及び戸籍附票システム標準仕様書【第3.1版】が取りまとめられましたので、公表します。
公表資料・ 住民記録システム標準仕様書【第5.1版】等の改正概要PDF
・ 住民記録システム標準仕様書【第5.1版】
・ 戸籍附票システム標準仕様書【第3.1版】
※ 住民記録システム標準仕様書【第5.1版】及び戸籍附票システム標準仕様書【第3.1版】については、以下の総務省ホームページにおいても公開します。
総務省トップ > 組織案内 > 研究会等 > 自治体システム等標準化検討会
https://www.soumu.go.jp/main_sosiki/kenkyu/jichitaishisutemu_hyojunka/index.html
The SIDI Hub Summit in Berlin, Germany, followed the 2023 inaugural event in Paris and was the second of 5 Summits in 2024. The five summits are critical forums for progressing the five workstreams throughout the year.
ドイツのベルリンで開催されたSIDIハブ・サミットは、2023年のパリでの初開催に続くもので、2024年に開催される5つのサミットの2番目となる。この5つのサミットは、年間を通じて5つのワークストリームを推進するための重要なフォーラムである。
5つのワークストリームはこれまでも書きましたが、以下の通りです。
Champion Use Cases Trust Framework Mapping Minimum Technical Requirements Governance Metrics of SuccessApproximately 50 attendees joined SIDI Hub Berlin on June 3, 2024 - the day before the European Identity and Cloud Conference (EIC). Due to the nature of who attends EIC, far more technical experts attended the Berlin Summit, whereas the Paris and Cape Town Summits included heavier representation from Governments. For this reason, the agenda emphasized naming barriers to interoperability, refining the approach for technical requirements gathering, and the interplay between Trust Frameworks and protocols. It also included a discussion on Champion Use Cases - particularly for the European context - and building a shared research agenda. The feedback from these sessions will shape workstream activities and the structure of Summits to follow. 欧州アイデンティティ・クラウド会議(EIC)の前日である2024年6月3日、SIDIハブ・ベルリンに約50名の参加者が集まった。パリとケープタウン・サミットでは政府関係者が多く参加したのに対し、EICでは技術専門家が多く参加した。 このため、アジェンダでは、相互運用性の障壁を挙げること、技術要件収集のアプローチを洗練させること、トラストフレームワークとプロトコルの相互作用が強調された。また、チャンピオンのユースケース(特に欧州の状況)についての議論や、 の共有研究アジェンダの構築も行われた。これらのセッションからのフィードバックは、ワークストリームの活動と、その後のサミットの構成を形作ることになる。
→レポートに書いている通り、European Identity & Cloud Conference(EIC)の前日に開催されたため、ケープタウンに比べると少し技術寄りだったかもしれません。また、アフリカに比べるとEUという一つの経済圏がすでに構築されていることがユースケースを考える上での特徴点となるのかもしれません。
This section synthesizes that information into the high-level themes that the co-organizers took away from the event. Part 2 of this report consolidates the Rapporteur’s notes and many of the slides from the event itself.
本セクションでは、これらの情報を統合し、共同主催者がこのイベントから持ち帰ったハイレベルなテーマを紹介する。本報告書の第2部では、報告者のメモとイベントでのスライドをまとめた。
アーキテクチャのタイプ
While there are vast numbers of configurations, profiles, and proprietary APIs related to the transmission of identity data, there are a limited number of architectural archetypes that need to be able to interoperate globally. In particular, those that are based on API connections between Identity Providers (IdPs) and Relying Parties (RP’s) and those that are mediated by an agent (“wallet”) that holds credentials issued to an end-user. Interventions to ensure interoperability of these solutions include options at the source (the IdP or the wallet) or at the destination (the Relying Party or Verifier). SIDI Hub participants remain in active discussion about whether there is a need for broad guidance about which interventions may be more appropriate or if the group should support both “fix-at-source” and “fix-at-destination” models.
ID データの伝送に関連する構成、プロファイル、および独自の API は膨大な数に上るが、グローバルに相互運用でき る必要があるアーキテクチャの原型は限られている。特に、ID プロバイダ(IdP)と依拠当事者(RP)間の API 接続に基づくもの、およびエンドユー ザに発行されたクレデンシャルを保持するエージェント(「ウォレット」)に仲介されるものである。これらのソリューションの相互運用性を確保するための介入には、ソース(IdP またはウォレット)またはデスティネーション(依拠当事者または検証者)におけるオプションが含まれる。SIDI ハブの参加者は、どの介入がより適切であるかについての広範なガイダンスが必要であるか、またはグループが「送信元での固定」と「宛先での固定」の両方のモデルをサポートすべきかについて、活発な議論を続けている。
→従来のIdPとRPのモデルに加えてウォレットが間に入るモデルを考える場合の相互運用性等に関する深掘りが必要になる、という話です。抽象度を上げていくと単純に送信者と受信者になりますが、それぞれの特性がアーキテクチャによって異なるため相互運用のためのプロトコルや信頼性を確保するためのトラストフレームワークの互換性なども考慮する必要があります。
Verifierがマーケットを作る
A significant theme of feedback at both SIDI Hub and the EIC event that followed was that Verifiers (in this paragraph, the term is used interchangeably with Relying Party) represent the more challenging side of the marketplace - and that adding complexity or risk to Verifier implementations may make it harder to do business. This would lead to:
1. Slow adoption and use
2. Barriers to entry in multiple verticals
3. Exacerbated inclusion issues for smaller players.
Any solution needs to address these risks. On the other hand, SIDI Hub participants are also keenly aware that once an ecosystem is functioning, verifiers stand to benefit more than the issuers, who often bear the cost of implementation. The group was eager to explore opportunities to address this imbalance.
SIDI ハブとそれに続く EIC イベントでのフィードバックの重要なテーマは、検証者(この段落で は、この用語を依拠当事者と同じ意味で使用する)が市場のより困難な側面を代表しており、検証者の実装に複雑性やリスクを加えると、ビジネスが困難になる可能性があるということであった。その結果、以下のことが起こる:
1. 導入と利用の遅れ
2. 複数の業種における参入障壁
3. 小規模プレーヤーのインクルージョンの問題の悪化。
どのような解決策も、これらのリスクに対処する必要がある。他方、SIDI ハブの参加者は、エコシステムがいったん機能すれば、多くの場合、導入コストを負担す る発行者よりも検証者の方がより多くの利益を得る立場にあることも強く認識している。グループは、この不均衡に対処する機会を探ることに熱心であった。
→この話もいつも議論になるところですね。ビジネスモデルをどうするか、という話です。現状ではやはりVerifierが一番便益を得ることができそうだ、ということなので基盤やサービスを開発するときは十分に年頭におく必要がありそうです。
ユースケースの根拠となる要件(あるいは「抽象化の危険性)
In exploring the set of 9 communication patterns between architectural archetypes, the Minimum Technical Requirements workstream concluded there was a limit to how much could reasonably be accomplished without the solid definition of a use case. As participants have pointed out, examples of successful cross-border interoperability efforts have been rooted in a specific problem space: passports for border crossings and air travel, credit card payments, a unified SAML profile for global access to educational institutions, etc. For this reason, the SIDI Hub Community continues its global listening and will work to define a set of 3-4 “Champion Use Cases” (and dive deeper into their respective requirements) throughout the next several Summits.
アーキテクチャ原型間の9つの通信パターンを検討する中で、最小技術要件ワークストリームは、ユースケースの確固とした定義なしに合理的に達成できることには限界があると結論づけた。参加者が指摘したように、国境を越えた相互運用性の取り組みが成功した例は、特定の問題空間に根ざしたものであった。たとえば、国境通過や航空旅行のためのパスポート、クレジットカード決済、教育機関へのグローバルアクセスのための統一SAMLプロファイルなどである。このような理由から、SIDIハブコミュニティはグローバルな聞き取り調査を継続し、今後数回のサミットを通じて、3~4件の「チャンピオンユースケース」の定義(およびそれぞれの要件への深掘り)に取り組む予定である。
→難しいところです。抽象化をしないと汎用的で互換性のあるシステムを構築することは難しくなりますが、ユースケース分析を進めれば進めるほど、「ユースケース依存」の話が大きくkなってくる、ということです。この辺りはアーキテクトの役割になるかと思います。
ブローカーとプロキシ
Given the expectation that these archetypes continue to exist because states make sovereign choices, it is prudent to assume that interoperability may depend upon brokers and proxies. While this undermines any attempt at a global, fully decentralized, and peer-to-peer architecture - and it necessitates careful consideration for privacy (i.e., masked data) and governance - it does not necessitate a centralized database or “phone home” architecture.
The terms “Broker” and “Proxy” should not be used interchangeably. Instead, the former may imply a specific market opportunity and commercial interest with pros and cons. Meanwhile, “Proxy” is intended to be a pass-through technical component and implies nothing about a business model.
国家が主権的な選択をするからこそ、こうした原型が存在し続けるという期待を考えれば、相互運用性はブローカーやプロキシに依存する可能性があると考えるのが賢明だ。このことは、グローバルで、完全に分散化された、ピアツーピアのアーキテクチャの試みを損なうものであり、プライバシー(すなわち、マスクされたデータ)とガバナンスに対する慎重な配慮を必要とするものであるが、集中化されたデータベースや「フォンホーム」アーキテクチャを必要とするものではない。
ブローカー 「と 」プロキシという用語は、同じ意味で使うべきではなく、前者は特定の市場機会を意味し、長所と短所のある商業的利益を意味する。一方、「プロキシ」はパススルーの技術コンポーネントを意図しており、ビジネスモデルについては何も示唆しない。
→システム構成を考えるとブローカーやプロキシを配置するモデルが安易で現実的な解になるのは想像に硬くないのですが、どうしても依存度が上がってしまう(場合によってはSPOFとなってしまう)ことはリスクとして捉えておかないといけないと思います。また、ブローカーとプロキシの違いはビジネス的な価値の創出が行われるかどうか、というところで区切りをつけた、ということもこれからのアーキテクチャの詳細化に向けて重要な合意事項だったと思います。
他にもトラストフレームワークや政府の関与のあり方などもまとめられていますが、この辺りは次回にでも。まずは技術にフォーカスした部分だけを先行してお届けしました。
Back in July, I made a post here in Medium on running Llama 3.1 8B locally not long after the model dropped. It is my most viewed and most read post by between 1 and 2 orders of magnitude based on my poor general reach at this writing. It was surely the timeliness and high interest in all things Llama 3.1 at that moment. I looked at the stats and they were enlightening on the distribution of referrals. Google is still king by a wide margin and my LinkedIn effort produced almost nothing.
Here are the stats:
The whole set was surprising to me. I made a post about this article only to LinkedIn so I was surprised at only one linkedin.com referral. I thought I had clicked on it more times than that. :-) It kind of makes me sad. Wow, Google. Not sure what I did to show up in some search results but I wish I did that more often. 146 in email, IM and direct. YCombinator… really. I’m honored. Gemini.google.com — interesting. Twitter — 2; very surprising as I didn’t post there and haven’t used Xitter in about a year for what would be obvious reasons for some people. Getliner.com — I don’t even know what that is. Sorry getliner.com. Identosphere.net — mildly surprised. I get picked up in the blog aggregator as I have in the past posted more reliably on digital identity and verifiable credentials.
This post shows how to implement an OpenID Connect back-channel logout using Keycloak, ASP.NET Core and .NET Aspire. The Keycloak and the Redis cache are run as containers using .NET Aspire. Two ASP.NET Core UI applications are used to demonstrate the server logout.
Code: https://github.com/damienbod/keycloak-backchannel
SetupThe applications are run and tested using .NET Aspire. The UI applications are setup using the OpenID Connect code flow with PKCE and OAuth PAR. Two docker containers are used, one for the Redis cache and one for the Keycloak server.
.NET Aspire SetupThe AppHost project in .NET Aspire is used to configure the different services. The Keycloak container is run using HTTPS with develop certificates. The ASP.NET Core applications are run using development certificates as well. For the Keycloak server to access the docker host, it must register the same developer certificates from the host, or disable the certificate trust manger inside the container. This works good, but should only be setup like this in development. The different project must reference each other as required. To use Redis and Keycloak, the Aspire Nuget packages for these containers need to be installed.
var keycloak = builder.AddKeycloakContainer("keycloak", userName: userName, password: password, port: 8080) .WithArgs("--features=preview") // for more details regarding disable-trust-manager see https://www.keycloak.org/server/outgoinghttp#_client_configuration_command // IMPORTANT: use this command ONLY in local development environment! .WithArgs("--spi-connections-http-client-default-disable-trust-manager=true") .WithDataVolume() .RunWithHttpsDevCertificate(port: 8081); var cache = builder.AddRedis("cache", 6379) .WithDataVolume(); var mvcpar = builder.AddProject<Projects.MvcPar>("mvcpar") .WithExternalHttpEndpoints() .WithReference(keycloak) .WithReference(cache); var mvcbackchanneltwo = builder.AddProject<Projects.MvcBackChannelTwo>("mvcbackchanneltwo") .WithExternalHttpEndpoints() .WithReference(keycloak) .WithReference(cache); Keycloak OpenID Connect client configurationThe Keycloak client should have the backchannel logout activated. The container uses the localhost applications from the docker host and so the host.docker.internal domain is used. The logout endpoint is implemented in the ASP.NET Core application.
ASP.NET Core LogoutEach ASP.NET Core application that supports the back-channel logout must have a server implementation and provide a web hook for the identity provider (Keycloak) logout event. If multiple instances are running, one can send a logout event to Keycloak. This ends the session on the identity provider and sends a logout post request to all server backends hosting the UI application. The logout event is handled and persisted to a distributed cache. For all other instances which request data from the server for the same user and session, the session is ended and the user must authentication again.
Logout request Webhook for logout event from Keycloak server for all instances Persist event to cache if missing and logout Logout UI on next HTTP request for user sessions services.AddTransient<CookieEventHandler>(); services.AddSingleton<LogoutSessionManager>(); services.AddHttpClient(); services.Configure<AuthConfiguration>(configuration.GetSection("AuthConfiguration")); var authConfiguration = configuration.GetSection("AuthConfiguration"); builder.AddRedisDistributedCache("cache"); services.AddAuthentication(options => { options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme; options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme; }) .AddCookie(options => { options.ExpireTimeSpan = TimeSpan.FromMinutes(60); options.Cookie.Name = "MvcPar"; options.EventsType = typeof(CookieEventHandler); }) .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options => { options.Authority = authConfiguration["StsServerIdentityUrl"]; options.ClientSecret = authConfiguration["ClientSecret"]; options.ClientId = authConfiguration["Audience"]; options.ResponseType = OpenIdConnectResponseType.Code; options.Scope.Clear(); options.Scope.Add("openid"); options.Scope.Add("profile"); options.Scope.Add("email"); options.Scope.Add("offline_access"); options.ClaimActions.Remove("amr"); options.ClaimActions.MapJsonKey("website", "website"); options.GetClaimsFromUserInfoEndpoint = true; options.SaveTokens = true; options.PushedAuthorizationBehavior = PushedAuthorizationBehavior.Require; options.TokenValidationParameters = new TokenValidationParameters { NameClaimType = JwtClaimTypes.Name, RoleClaimType = JwtClaimTypes.Role, }; });Note: The sample code in the repository was created using the IdentityServer4 Samples.
Redis cacheRedis Insight can be used to view the Redis cache data. Each time the application handles a new user and session logout event, it persists the event to the cache. If any further application instances are authenticated using this session and user, the application will sign-out as well on the next server event. The expiry time of the cache should be long enough so that an offline session cannot be opened after it expires.
LimitationsThe back-channel logout only works on a per browser session because Keycloak creates new sessions for each browser. When the logout is received it is filtered and handled using the Keycloak session. If Keycloak can use a single session for all browsers of each user, then the logout can work for all active UI apps of the same user.
Linkshttps://openid.net/specs/openid-connect-backchannel-1_0.html
https://ldapwiki.com/wiki/OpenID%20Connect%20Back-Channel%20Logout
https://datatracker.ietf.org/meeting/97/materials/slides-97-secevent-oidc-logout-01
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/app-state
https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/cache-dotnet-core-quickstart
https://github.com/RedisInsight/RedisInsight
I recently posted on my experience with using a few Python examples of retrieval augmented generation (RAG) with an LLM (OpenAI API). They were underwhelming, though they did provide a lot of insight for me. This post is about my next experiment: let’s get AI tools to generate examples for me. The results were mixed but not for reasons you might think. Here’s what I found using ChatGPT, Microsoft Copilot and Cursor.
Note: you can find the generated code in my LLM_Generated_RAG Github repo. I will likely add to this repo as I try new prompts, LLMs, and approaches.
ChatGPT 4oLet’s start with the most well-known, ChatGPT. I used a simple prompt to see what it would do. I didn’t provide prompts for different parts of the code. It should not be very extensive code so I didn’t feel the need to divide it into separate prompts. If you want to see my very elementary prompt pull up the ChatGPT related file in the repo.
The first thing I noticed is that it was using PyPDF2 which was deprecated in Decemeber of 2022. Not surprising as, almost certainly, a lot of what the model was trained on used PyPDF2. It actually created well-structured code with functions for the different discrete operations. What I realized later, due to the output of the Cursor tool, is that it created a more granular set of code than it had to. This is exactly what I was looking for but didn’t specify in the prompt. What do I mean? It did the following:
read the PDF and extracted the text split the document into chunks created embeddings for the chunks using the OpenAI text-embedding-ada-002 created an embedding for the query searched the chunks using faiss generated a response using the query and the chunk(s) as contextThis is what I wanted. Turns out there was a very different way to interpret the prompt.
The downside. The code didn’t run out the box. Also, I believe it only used one chunk for the context to send along with the query. I have to investigate the code more closely to be sure on that single chunk thing. The embedding search chunk function I need to investigate more to understand it. I didn’t try to make the code run for now as that was not part of the experiment. I expect I will try to modify it soon to function. I’ll report back the results.
This granular set of actions was very different than what I got from Cursor.
CursorThe Cursor prompt was not exactly the same as what I used for ChatGPT, which was bad of me to do. I got a little lazier, but the crux was the same. You can see that prompt here.
The results were very different. It did not provide the granular steps that ChatGPT did. It met the requirement more succinctly.
extract the text from the PDF, also using PyPDF2 pass the entire text to ChatGPT as context along with the promptWhen I first posted this I said it rain out of the gate, but that was wrong. It suffered from the same issue all three code generations did. It tried to use an old call of openai chat completion. With that fixed I believe it would, hypothetically, return the expected results. It would work similarly to how one does it in the ChatGPT interface. Not what I wanted, but I hadn’t specified to break it down so that I was creating embeddings etc. Fair game.
The downside, the context was too large (43,000+ tokens) for the tokens per minute limit of my OpenAPI account level and gpt-4o model (see attached note for an explanation). So I didn’t get to see the output, but have no doubt it would have produced similar results as if I had done it through the ChatGPT user interface.
Microsoft CopilotWhat can I say? I don’t know what I would have gotten here because Copilot blocked the output. This is apparently because “GitHub Copilot checks code completion suggestions with their surrounding code of about 150 characters against public code on GitHub. If there is a match, or a near match, the suggestion is not shown to you.”
There is a flag you can set in your Github settings to turn this on or off. I checked and my flag and it is set to “Allowed” but you see the results.
I’ll continue to try to troubleshoot this issue and see if I can get something out of Copilot and update this post if I do.
In recapIn recap. One provided the detail I wanted even though I hadn’t specified it, a happy coincidence, but it did not run out of the gate. The second took a very different approach and would have provided the desired results in LLM response if my OpenAPI account level had supported a large enough tokens per minute for 43,000+ tokens in the context. But it wasn’t the code I was hoping for. Copliot just didn’t work for me for reasons I don’t yet understand. More experimentation to come.
If a set of accounts live in isolation forever, what happens to those accounts only matters within that isolated system. But when we make a bubble in our federated network, we aren’t just making a stand-alone system that can go live in a fully disconnected state. Over time, we expect things to re-connect, and when that happens, data needs to be harmonized across the boundaries.
So many bubbles, so many connections Data SynchronizationWhen multiple independent systems live together in the world and share data, inevitably that data is going to get out of sync. In a purely heirarchical system, we’re mostly concerned with building a consensus around the correct state of the shared data. We see this approach in distributed ledger systems, where nodes eventually come to a common view of what the shared state of the world ought to be.
But we don’t have that in a bubble-based architecture, because we don’t expect everyone to have the same view. Instead, we expect many different independent views to contribute to each other in a distributed fashion. This effectively means that for each bubble, it can send updates in and out to other systems. In most cases, there is a directionality to the data flow: one side is going to be treated as more authoritative than the other for a given context. What a bubble does when it’s on either end of that gap changes how we view the synchronization.
Account Updates From AboveWhen changes are pushed to us from an authoritative source, the simplest thing is to overwrite everything. After all, if it’s authoritative, why wouldn’t we just take that statement as truth? But the reality is that we’ve likely augmented our record of that user with additional details, overrides, and other localized changes that we don’t want to lose.
Local data (blue) shadows updates from the source (orange), in some casesIn these cases, we can shadow the data. In other words, we keep a copy of the source’s data separate from our own local view. When we get an update from that source, we can update our copy of the source’s data with wild abandon. We can then decide, by local policy, whether we want to adjust our overrides based on the update. Importantly, this decision is separate from accepting and processing the updated data from the source. This setup allows us to keep local information in the bubble at the same time that we sync from elsewhere.
Account Updates From BelowAn authoritative bubble is sometimes going to want to pull updated information from the bubbles that live downstream. These are systems that we’ve sent accounts out to, and those systems might have something to tell us about our users. Maybe there’s an additional accreditation that’s been earned, or something to represent additional access details outside of our bubble, or even just an update to one of the core fields we sent down.
In any case, the downstream bubble is sending us additional data about a user, and we now have the chance to do something about it. If nothing else, we can store it and note it. If we want to, we can update the user’s record that we hold locally, and even go so far as to propagate that downward again to other bubbles.
Changing TrustIt’s not just user data that we can pass around, though that’s the most common piece we’d expect to see. The bubbles can also pass about other bubbles to each other, and incorporate what they learn into their own systems.
For many situations that fit the bubble architecture patterns, we expect to meet new peers in the field and make new decisions based on local context and requirements. These newly-discovered bubbles can then be propagated through the network, along with potentially interesting information about what the presenting bubble trusts it for.
Audit LogsNow that we can identify users and the systems they come from, we can start to do one of the most interesting and perhaps complex jobs of a reconnected bubble: audit logs. When a bubble gets provisioned, that provisioning authority is likely to want to know what happens in that bubble during the disconnected spells. The bubble can package up the relevant audit log history and pass it along to the authorities that need it.
But auditing can go further than that: for any user that enters our bubble from somewhere else, we probably want to report back to the system that sent them to us. Since we know where we originally learned of them, and we know how to identify that system as a whole, we can filter and target the information we’re sending over. And we can do this while processing the updates they’re sending us about the account.
ConclusionThe bubble pattern sits between the always-connected and always-separate models, but the moment of reconnection gives us a profound opportunity to process user and trust data in ways that we may have never considered.
The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.
Today, we cover:
Industry pulse. Microsoft Recall to be unremovable, another “AI talent raid” by Amazon, Ticketmaster struggles to handle common load challenges – sometimes, rise and fall of OpenSea, and more.
More trad tech companies to move off public cloud? In a recent report, 83% of chief information officers say they want to move more workload to private clouds. Cloud computing has matured and is becoming more expensive, while private cloud technology is increasingly accessible.
CockroachDB: open source product goes proprietary. Another VC-funded open source product closes itself off, in hope of generating more revenue and profits. Many users need to open their wallets and buy annual licenses, or fork and maintain an older version of CockroachDB.
Software glitch almost floods Amsterdam. Dams near the Dutch capital that were supposed to be closed, mysteriously opened at night during a violent storm. Disaster was averted by a curious employee – and the workaround to avoid similar incidents includes 24-hour human monitoring of dams and water levels.
1. Industry pulse Microsoft Recall to be unremovableMicrosoft’s struggle with its controversial Recall feature continues. Recall is a “photographic memory” feature planned for Windows 11 which takes constant screenshots of a device, and makes it searchable. Its launch in June was a disaster, as Microsoft skipped basic security and privacy approaches – making these screenshots easily accessible for bad actors. A week after, Microsoft “recalled” Recall, confirming that it will not ship it with these glaring holes.
Now, the Redmond-headquartered tech giant is doubling down on Recall: it will be shipping on Windows 11, and the feature will have no option to be uninstalled. Users won’t be able to disable it.
The problem with making Recall core to Windows with no way to remove it or turn it off at the OS level, is that it could become a dealbreaker for enterprise clients which will not want an invasive, potentially insecure feature to run on machines by default. It’s also strange to see Microsoft so set on shipping this feature that is clearly designed to make some use of the hardware GPUs, even though the utility is still questionable. The company will surely keep getting bad press, and privacy-focused techies will have one more reason to stay clear of Windows.
Another “AI talent heist”“Communities that want to build comprehensive public calendars will be able to do so using a hybrid approach that blends existing iCalendar feeds with feeds synthesized from web calendars. It’s not a perfect solution, but with LLM assistance it’s a workable one. And who knows, maybe if people see what’s possible when information silos converge, the common tools that can ease convergence will seem more attractive.” — An LLM-Turbocharged Community Calendar Reboot
Part of the LLM series at The New Stack.
At Bar del Pla in Barcelona
I have been wanting to experiment with creating some code performing retrieval augmented generation (RAG) for an LLM. So far I am 0 for 2. In both examples I tried, that were professed to work, it fell far short. I blame embedding and retrieval. I also blame myself… who else is there to blame. I know RAG works in some form because I can go onto ChatGPT upload a PDF and prompt about it with expected and quality results; but, when trying to replicate similar outcomes from code and the OpenAI API I get disappointing results. Here’s the story.
Experiment one was part of a Udemy course on RAG and mastering some better approaches than naive RAG. Specifically RAG with query expansion. It was a very short course but had a couple of cohesive Python examples that built out over the course material. It included the document to use for the augmentation. Additionally, unlike experiment two, it largely didn’t rely on a ton of abstraction of the underlying basic concepts. After reading and parsing the text from the PDF I used RecursiveCharacterTextSplitter and
SentenceTransformersTokenTextSplitter from Langchain to create my chunks. Added my chunks to Chroma db in memory using their default embedder. Took my query and performed the retrieval of 5 chunks from the Chroma db. No reranking performed. The returned chunks were at best partial results and at worst just not as good as expected. What I hadn’t noticed about the code from the instruction when I went through the class is that it never passed the chunks back as context to the LLM the second time along with the query to get an answer. Kind of an important part to be missing. I can tell from the returned chunks it would not have produced a satisfactory answer had the closing action before performed. I tried with differing chunk sizes and overlaps and never received better results. I tried with my own document and faired no better. I chalked it up to a disappointing and poor example. Perhaps this is why it didn’t go the final step to pass it all back to the LLM for a response.
I moved on to a second experiment that used a bit more abstraction by relying on Langchain significantly more. It was also doing naive RAG, not augmenting the prompt from the LLM initially in any way. This time it did have the pass to the LLM in the end to get the LLM response. Looking at the chunks it retrieved I could tell I wasn’t going to get a satisfactory response. I had asked it for a list of 10 principles that were specifically outlined in the document in a single paragraph. The best in all my attempts was that I got 2, maybe 3 of the list. Again I played around with chunk size and overlap and generally only got worse results. The results in the demonstration had not seemed much better when I evaluated it more closely.
All in all I need a third+ experiment. I am trying to get a response even remotely similar to what I get from the ChatGPT. Obviously, my examples are not high enough quality.
Do you have such an example?
Time to ask ChatGPT, Claude, Github Copliot, or Cursor — sounds like a job for Cursor — to create a python program for me and see how it functions.
Q: “I’d like to make a better case for paying down tech debt on my team. What are some proven approaches for this?”
The tension in finding the right balance between shipping features and paying down accumulated tech debt is as old as software engineering. There’s no one answer on how best to reduce tech debt, and opinion is divided about whether zero tech debt is even a good thing to aim for. But approaches for doing it exist which work well for most teams.
To tackle this eternal topic, I turned to industry veteran Lou Franco, who’s been in the software business for over 30 years as an engineer, EM, and executive. He’s also worked at four startups and the companies that later acquired them; most recently Atlassian as a Principal Engineer on the Trello iOS app.
He’s currently an operating partner at private equity firm SilverTree Equity, and technical co-founder at a new startup. Lou says he isn’t delighted about the volume of tech debt accumulated during his career, but is satisfied with how much of it he’s managed to pay down.
In this guest post, Lou shares insights about how to approach tech debt. Later this year, he’s publishing a book on the subject. For updates on this upcoming release, subscribe here.
In this article, we cover:
Pay down tech debt to go faster, now. It’s common for less experienced engineering leaders to assume that focusing on features and ignoring tech debt is how to move faster. Lou used to agree, but not any more.
Use tech debt to boost productivity. Three examples of when tackling tech debt empowered engineering teams to move faster, right away.
Link tech debt and value delivery. When tech debt is tackled sensibly, it delivers business value. At Trello and Atalasoft, this was how Lou got engineers and management to appreciate the task.
Make tech debt’s effects visible. Dashboards are an effective way to visualize the impact of tech debt. A few examples.
Factor in time saved when thinking about productivity. Reducing tech debt typically improves coding, reviewing, and deployment for everyone.
Use tech debt payments to get into the flow. In a counter-intuitive observation: by making small, non-functional improvements, you gain more confidence in a new codebase, and can start to move faster.
Big rewrites need heavyweight support. Without the backing of management, a large-scale rewrite is likely to fail.
With that, it’s over to Lou:
1. Pay down tech debt to go faster immediatelyWhat is tech debt?
I define tech debt as any problem in the codebase that affects programmers by making it harder to make necessary changes. As a programmer, I wanted to fix such issues because they slowed me down. But as a manager, I had to ensure the team delivered value to stakeholders. I’ve gone back and forth between these roles during my career, and made mistakes in both directions – but I also learned a lot about getting the balance right.
Reducing tech debt pays off immediately in faster buildsIn 2010, I was head of development at Atalasoft, a company in the .NET developer tools space. I was obsessed with shipping, and spent all my time worrying about delivering the features in our roadmap. Over time, we improved at this, which showed up in our revenue growth and eventually led to an acquisition.
We were in a competitive market with more opportunities than we could handle. We had fewer than ten developers, but we were bootstrapped, so had to stay profitable and couldn’t just hire ahead of revenue.
The stakes got even higher after we were acquired. We had two years to deliver on an ambitious roadmap, for which there was an earnout bonus contingent upon delivery. If we didn’t deliver, we’d likely be classified as a failed acquisition. Our new owners had just had one such failure, which led to downsizing and an office closure.
My mindset was that any time spent on technical debt meant we’d fail to deliver on the roadmap. We couldn’t risk the deadline by wasting time cleaning up old messes, and had to choose between tech debt and roadmap delivery. In hindsight, I see this was wrong. I learned that the mindset of ignoring tech debt hurt my team.
Following an exit interview in which a departing engineer mentioned tech debt had contributed to their decision to leave, I started asking people during 1:1s how they felt about it. Their responses showed their frustration with me.
By then, I had been a developer for over fifteen years, and had worked in codebases with a lot of debt, so I knew what it was like. But by now, I was a manager who sometimes contributed code, but had forgotten what it was like to be thwarted by the codebase every day. To my team, I was part of the problem. They had been trying to tell me this, but I didn’t get it. Their departing colleague with nothing to lose in their exit interview finally got through to me and helped me understand the depth of the issue, and that it was slowing everyone down.
I learned an important lesson: the cost of tech debt is borne daily by your team, and you risk damaging motivation and raising attrition by ignoring it. Even if you have every reason to move forward without addressing tech debt, being an empathic manager requires you at least do something. Doing nothing – like I did – is not a good option.
So we started making changes. The biggest problems were with our build system and installer because they affected every developer and every product. It was a tangled bag of legacy code that needed constant maintenance, but it wasn’t very big, and I approved a plan to rewrite it with modern tools. It was a small experiment but paid off right away in quicker CI builds and an easier-to-modify codebase. Most importantly, I saw it didn’t derail our roadmap, so we took on other small initiatives.
This taught me another lesson about addressing technical debt. I had thought of it as something that might pay off in the long run. Might. This belief made it hard to justify doing it when I had to deliver on short-term goals. But instead, something else happened:
We paid off tech debt and increased productivity instantly! We had a build with faster feedback loops, less cognitive load, and which didn’t make developers frustrated when they had to add to it, which happened regularly. Updates were made with less code and without breaking things. It was an example of tech debt reduction paying off in increased developer productivity, right away.
Learning the cost of too much rewriting at TrelloI got my next lesson at Trello where I worked on the iOS app. The codebase was three years old when I joined in 2014. It had understandable tech debt because they needed to move fast, after going from 0 to 6 million sign ups. The devs working on it were founding engineers, working as described by The Pragmatic Engineer in Thriving as a Founding Engineer, and seeking product-market fit. Our biggest tech debt issue were some frameworks that made it fast to build a simple app, but held us back as the app got more complex.
Our own choices were influenced by the speed of Apple’s updates to iOS. The iOS 7 update completely changed the iOS design language and its network APIs. Later, iOS 8 introduced presentation controllers that gave developers much control over the animation when new views are shown. Unfortunately, the iOS 8 change broke our navigation code and caused crashes. These added up and started to make our code seem antiquated.
Left: how most iOS 6 apps looked like. Right: the iOS 7 version. iOS 7 introduced a ‘flatter’ look, and required lots of code changes to implement this new UI. Screenshot is from my app called HabitsOur code got even more complex when Apple decided to feature Trello on physical iPhones at Apple Stores. To be in stores, we needed a build that worked without an account or a network, so a mock backend was embedded in it for demo purposes. We didn’t want to maintain a separate codebase, so had a lot of random bits of demo-mode logic that stayed for years.
At Trello, I was coding every day and all this was in my face. Luckily, we were a small team of three developers, so my direct manager was also coding every day and was empathetic to the problems.
We did rewrites as we went, but sometimes went too far. To deal with the presentation controller problem of iOS 8, we developed a new paradigm for screen navigation inside the app, and rewrote all navigation to use it. This approach was the exact opposite of what I did at Atalasoft, where I’d ignored all tech debt.
Unfortunately, the approach of rewriting early turned out to be overkill. In hindsight, we could have just corrected the places that had crashed, and then lived with the code we had. Instead, we spent a few months designing and implementing a new, non-standard way of writing navigation code, but forgot a vital lesson that one of our founders, Joel Spolsky, identified in 2000 in Things You Should Never Do:
“We’re programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.
There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:
It’s harder to read code than to write it.”
On the Trello engineering team, we were all very familiar with this article and quoted it to each other often, but it still sometimes bit us. The urge to rewrite a system instead of fixing it is strong, and we couldn’t resist! We should have addressed the few complex navigation cases that crashed our code without the full rewrite.
Size tech debt payment to be proportional to value. This is the biggest lesson I learned on this project.
I’ve seen both extremes of dealing with tech debt:
As a manager, I was overly resistant to devoting time to dealing with technical debt
As an engineer, I was exposed to its problems every day and didn’t resist the urge to pay it off enough
These two extremes form the fundamental tension of dealing with tech debt. As usual, there needs to be a balance, but finding it is not so easy.
The heuristic I use to pay tech debt these days is this: by reducing a specific tech debt, can I increase developer productivity and deliver business value right now?
If I can’t, then I don’t pay it down.
When the debt is so big that it couldn’t possibly deliver value now, or the value is invisible so nobody sees it, I do something else. Let me break down my heuristic…
2. Use tech debt to increase productivityI try to pay down a little bit of tech debt regularly by making small cleanup commits as I go. I started doing this more intentionally after reading Kent Beck’s book, Extreme Programming Explained, in 1999, which introduced me to automated unit tests and continuous integration. Then, when I read Martin Fowler’s Refactoring, I started to see how to improve a codebase over time with very small, behavior-preserving changes checked by unit tests. In both books, and in others like Working Effectively with Legacy Code by Michael Feathers, and Kent Beck’s recent, Tidy First?, the authors stress that technical debt is inevitable, and that the main way to curtail it is to be constantly fixing it with small improvements enabled by unit tests and mechanical refactoring. I agree.
Unit tests, refactoring, and continuous integration are ubiquitous in the kinds of software I write, which are B2B SaaS productivity applications. Even making small improvements on an ongoing basis is common among my coworkers. It doesn’t take long, and there are usually quick wins to be had, like making the code more readable, or using a unit test to show how the code is supposed to work. Even in frontend code, Trello iOS adopted Model-View-ViewModel (MVVM) so we could test view-logic. We got the immediate productivity benefit of being able to run view code repeatedly without needing to manipulate a running app through several screens to check that our changes worked.
The issue is when the debt is large, which is where I struggled. My problem at Atalasoft was not with small improvements; it was with the bigger ones where I’d have to trade off current benefits like delivering features for the roadmap, for uncertain future benefits.
But I realized something.
You can get productivity benefits immediately, even with larger initiatives. If you do it right, you will deliver feature work faster and with higher quality. In fact, I view tech debt proposals that don’t deliver instant developer productivity gains as suspect.
Rewriting the build and installer at Atalasoft brought an immediate productivity boost. We had a backlog of problems and new additions, but the rewrite took one developer around a month, and when it was done many of the problems just went away because the new system was based on a framework wherein many problems could not occur, meaning we could close a bunch of reported bugs. The new system was unit testable, so we didn’t need to build and install the entire system during development to test our new changes while we were writing it. We also got more benefits later, but the instant benefits justified it.
At Trello, adding unit tests to a codebase helped me finish a project faster. When I joined in 2014, we were just about to start our internationalization (i18n) project, which I took on for the iOS app. One part was to write support for i18n-safe string interpolation (inserting variables or expressions into a string,) and pluralization (adjusting words to plural forms based on the number they refer to, to make the phrases grammatically correct) – which was only partially supported in iOS at the time. It’s standard string manipulation at its core, but in 2014 our iOS app didn’t have unit tests.
Without unit tests, if I had wanted to run the code, I’d need to run the app and then tap-tap-tap until I got to a specific string. I would have to do this for each kind of string I generated. But with unit-tests, I could just list all the examples with their expected results, and run tests in less than a second. So, I proposed to the team to add unit tests to our build and CI.
No one was against unit tests, but it hadn’t been a priority. Most of the code was UI or network code, for which unit tests are harder to write. But the code I was writing was highly testable, and in fact, it’s harder to write without tests. So, I added the unit test project to our workspace and wrote the string code. With the unit test project there, the other developers added tests to their work. I was there for six more years, and saw the benefits of the tests over time, especially in complex code like our sync engine. But that’s not why I did it: I added the unit tests to go faster immediately.
Also at Trello, creating an abstraction layer for the design system made us more productive. Eventually, we created a design system with a reduced set of fonts, colors, and other design attributes and specific rules for using them. Before, it was common to see hardcoded values in view controllers throughout the app, as each screen implemented the designer’s specification for that screen, which wasn’t always consistent. We could have just updated those lines to the new approved values, but it was the perfect time to make an abstraction for the design system itself. Doing this made it faster to write code that matched a design, and when a default in the design system changed, it would be reflected everywhere.
These three examples also adhere to another heuristic I use for finding the right balance with tech debt: coupling it with delivery of value.
3. Couple tech debt fixes with value deliverySchon morgen früh sollen die EU-Regierungen den umstrittenen Gesetzentwurf zur Chatkontrolle, der im Juni nach massiven Protesten von der Tagesordnung genommen wurde, jetzt doch unterstützen. Erreichen will die neue ungarische Ratspräsidentschaft dies nach einer Meldung des Nachrichtendienstes Contexte mit einer von Politico geleakten Detailänderung, nämlich dem Verzicht auf die besonders fehleranfällige Suche nach unbekanntem Material im Zuge der verpflichtenden Chatkontrolle (freiwillig soll sie möglich bleiben). Geplant bleibt aber ansonsten die automatisierte Durchsuchung und gegebenenfalls Ausleitung privater Chats, darunter auch Ende-zu-Ende-verschlüsselte Chats, die zuvor klassifizierte verdächtige Fotos oder Videos enthalten sollen. Lehnt ein Nutzer diese „Upload-Moderation“ seiner Chats ab, könnte er keinerlei Bilder, Videos oder URLs mehr senden oder empfangen. Signal und Threema haben angekündigt, dass sie ihre Dienste in der EU einstellen würden, wenn sie gezwungen würden, die vorgeschlagene automatisierte Massenüberwachung (sogenanntes „Client-Side Scanning“) umzusetzen.
Der ehemalige Europaabgeordnete der Piratenpartei, Patrick Breyer, ruft die Bürger jetzt dazu auf sich an die EU-Regierungen zu wenden: „Im Juni gab es unter massivem öffentlichen Druck noch eine hauchdünne Sperrminorität zur Rettung des Digitalen Briefgeheimnisses, aber kaum bemerkte Blitzaktionen, zwischenzeitliche Regierungswechsel und Minimalzugeständnisse können das schnell ändern. Falls die Chatkontrolle kommt, werden wir gängige sichere Messenger ganz einfach nicht mehr nutzen können – das bedeutet wir verlieren den Kontakt zu unseren Freunden und Kollegen in der ganzen Welt“, warnt Breyer. „Wollen wir wirklich, dass Europa weltweit führend beim Abhören unserer Smartphones und der flächendeckenden Überwachung der Chats von Millionen gesetzestreuer Bürger wird? Das Europäische Parlament ist überzeugt, dass diese orwellsche Chatkontrolle das dringliche Anliegen eines besseren Kinder- und Opferschutzes verrät, weil sie unweigerlich vor Gericht scheitern wird. Es fordert deshalb einen wirklich wirksamen Kinderschutz durch sicherere Apps, proaktive Säuberung des Internets und eine Pflicht zur Löschung illegaler Inhalte – nichts davon ist in dem dem neuesten Orban-Vorstoß vorgesehen, zu dem sich die Regierungen morgen positionieren sollen. Jetzt liegt es an uns ihn zu stoppen!“
Die Chatkontrolle auf vermeintlich „bekannte“ illegale Inhalte beschränken zu wollen, bezeichnet Breyer als Augenwischerei: „Egal mit welchem Ziel – auch die Post darf nicht einfach jeden Brief verdachtslos öffnen und durchschnüffeln. Gerade die von US-Konzernen schon heute freiwillig praktizierte Durchleuchtung nach vermeintlich bekannten Inhalten führt zur Ausleitung tausender völlig legaler privater Chats, zur Überlastung der Strafverfolger und zur massenhaften Kriminalisierung Minderjähriger. Massenüberwachung ohne jeden Anlass zerstört das digitale Briefgeheimnis und sichere Verschlüsselung, auf die wir alle und auch unsere Kinder dringend angewiesen sind.“
Breyers Infoportal zur Chatkontrolle: chatkontrolle.de
Hilf jetzt mit die Chatkontrolle zu stoppenNIST SP800-63-4 2pd を読む会(2) を9月6日(金)午後9時からやりたいと思います。 今回は、無印のアイデンティティモデルとパートCを合わせて、フェデレーションとウォレット周りについて読んでいきたいと思います。前回に引き続き、勝原さんにもご参加いただける見込みです。 奮ってご参加ください。
前回をご覧になっておられない方は、こちらのブログと前回の動画も予習しておくと良いかも知れません。
【関連文書】
PDF版
SP 800-63 Revision 4 SP 800-63A Revision 4 SP 800-63B Revision 4 SP 800-63C Revision 4テキスト版 (github)
https://pages.nist.gov/800-63-4/sp800-63.htmlThis post shows how to implement an ASP.NET Core application which uses OpenID Connect and OAuth PAR for authentication. The client application uses Keycloak as the identity provider. The Keycloak application is hosted in a docker container. The applications are run locally using .NET Aspire. This makes it really easy to develop using containers.
Code: https://github.com/damienbod/keycloak-backchannel
SetupThe standard Aspire Microsoft template was used to setup the .NET Aspire AppHost, ServiceDefaults projects. The Keycloak container service was added to the AppHost project using the Keycloak.AuthServices.Aspire.Hosting Nuget package. An ASP.NET Core Razor Page project was added as the UI client, but any project can be used like Blazor or an MVC application.
Keycloak SetupThe Keycloak Container is completely setup in the AppHost project. The Keycloak.AuthServices.Aspire.Hosting Nuget package is used to add the integration to .NET Aspire. For this to work, Docker Desktop needs to be installed in the development environment. I want to use the Keycloak preview features and initialized this using the WithArgs method. If using the Microsoft Keycloak package, the setup is almost identical.
var userName = builder.AddParameter("userName"); var password = builder.AddParameter("password", secret: true); var keycloak = builder.AddKeycloakContainer("keycloak", userName: userName, password: password, port: 8080) .WithArgs("--features=preview") .WithDataVolume() .RunWithHttpsDevCertificate(port: 8081);I want to develop using HTTPS and so the Keycloak container needs to run in HTTPS as well. This was not so simple to setup, but Damien Edwards provided a solution which works great.
The RunWithHttpsDevCertificate extension method was added using his code and adapted so that the port is fixed for the HTTPS Keycloak server. This implementation requires the System.IO.Hashing Nuget package.
using System.Diagnostics; using System.IO.Hashing; using System.Text; namespace Aspire.Hosting; /// <summary> /// Original src code: /// https://github.com/dotnet/aspire-samples/blob/b741f5e78a86539bc9ab12cd7f4a5afea7aa54c4/samples/Keycloak/Keycloak.AppHost/HostingExtensions.cs /// </summary> public static class HostingExtensions { /// <summary> /// Injects the ASP.NET Core HTTPS developer certificate into the resource via the specified environment variables when /// <paramref name="builder"/>.<see cref="IResourceBuilder{T}.ApplicationBuilder">ApplicationBuilder</see>. /// <see cref="IDistributedApplicationBuilder.ExecutionContext">ExecutionContext</see>.<see cref="DistributedApplicationExecutionContext.IsRunMode">IsRunMode</see><c> == true</c>.<br/> /// If the resource is a <see cref="ContainerResource"/>, the certificate files will be bind mounted into the container. /// </summary> /// <remarks> /// This method <strong>does not</strong> configure an HTTPS endpoint on the resource. Use <see cref="ResourceBuilderExtensions.WithHttpsEndpoint{TResource}"/> to configure an HTTPS endpoint. /// </remarks> public static IResourceBuilder<TResource> RunWithHttpsDevCertificate<TResource>(this IResourceBuilder<TResource> builder, string certFileEnv, string certKeyFileEnv) where TResource : IResourceWithEnvironment { const string DEV_CERT_DIR = "/dev-certs"; if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Export the ASP.NET Core HTTPS development certificate & private key to PEM files, bind mount them into the container // and configure it to use them via the specified environment variables. var (certPath, _) = ExportDevCertificate(builder.ApplicationBuilder); var bindSource = Path.GetDirectoryName(certPath) ?? throw new UnreachableException(); if (builder.Resource is ContainerResource containerResource) { builder.ApplicationBuilder.CreateResourceBuilder(containerResource) .WithBindMount(bindSource, DEV_CERT_DIR, isReadOnly: true); } builder .WithEnvironment(certFileEnv, $"{DEV_CERT_DIR}/dev-cert.pem") .WithEnvironment(certKeyFileEnv, $"{DEV_CERT_DIR}/dev-cert.key"); } return builder; } /// <summary> /// Configures the Keycloak container to use the ASP.NET Core HTTPS development certificate created by <c>dotnet dev-certs</c> when /// <paramref name="builder"/><c>.ExecutionContext.IsRunMode == true</c>. /// </summary> /// <remarks> /// See <see href="https://learn.microsoft.com/dotnet/core/tools/dotnet-dev-certs">https://learn.microsoft.com/dotnet/core/tools/dotnet-dev-certs</see> /// for more information on the <c>dotnet dev-certs</c> tool.<br/> /// See <see href="https://learn.microsoft.com/aspnet/core/security/enforcing-ssl#trust-the-aspnet-core-https-development-certificate-on-windows-and-macos"> /// https://learn.microsoft.com/aspnet/core/security/enforcing-ssl</see> /// for more information on the ASP.NET Core HTTPS development certificate. /// </remarks> public static IResourceBuilder<KeycloakResource> RunWithHttpsDevCertificate(this IResourceBuilder<KeycloakResource> builder, int port = 8081, int targetPort = 8443) { if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Mount the ASP.NET Core HTTPS development certificate in the Keycloak container and configure Keycloak to it // via the KC_HTTPS_CERTIFICATE_FILE and KC_HTTPS_CERTIFICATE_KEY_FILE environment variables. builder .RunWithHttpsDevCertificate("KC_HTTPS_CERTIFICATE_FILE", "KC_HTTPS_CERTIFICATE_KEY_FILE") .WithHttpsEndpoint(port: port, targetPort: targetPort) .WithEnvironment("KC_HOSTNAME", "localhost") // Without disabling HTTP/2 you can hit HTTP 431 Header too large errors in Keycloak. // Related issues: // https://github.com/keycloak/keycloak/discussions/10236 // https://github.com/keycloak/keycloak/issues/13933 // https://github.com/quarkusio/quarkus/issues/33692 .WithEnvironment("QUARKUS_HTTP_HTTP2", "false"); } return builder; } private static (string, string) ExportDevCertificate(IDistributedApplicationBuilder builder) { // Exports the ASP.NET Core HTTPS development certificate & private key to PEM files using 'dotnet dev-certs https' to a temporary // directory and returns the path. // TODO: Check if we're running on a platform that already has the cert and key exported to a file (e.g. macOS) and just use those instead. var appNameHashBytes = XxHash64.Hash(Encoding.Unicode.GetBytes(builder.Environment.ApplicationName).AsSpan()); var appNameHash = BitConverter.ToString(appNameHashBytes).Replace("-", "").ToLowerInvariant(); var tempDir = Path.Combine(Path.GetTempPath(), $"aspire.{appNameHash}"); var certExportPath = Path.Combine(tempDir, "dev-cert.pem"); var certKeyExportPath = Path.Combine(tempDir, "dev-cert.key"); if (File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { // Certificate already exported, return the path. return (certExportPath, certKeyExportPath); } else if (Directory.Exists(tempDir)) { Directory.Delete(tempDir, recursive: true); } Directory.CreateDirectory(tempDir); var exportProcess = Process.Start("dotnet", $"dev-certs https --export-path \"{certExportPath}\" --format Pem --no-password"); var exited = exportProcess.WaitForExit(TimeSpan.FromSeconds(5)); if (exited && File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { return (certExportPath, certKeyExportPath); } else if (exportProcess.HasExited && exportProcess.ExitCode != 0) { throw new InvalidOperationException($"HTTPS dev certificate export failed with exit code {exportProcess.ExitCode}"); } else if (!exportProcess.HasExited) { exportProcess.Kill(true); throw new InvalidOperationException("HTTPS dev certificate export timed out"); } throw new InvalidOperationException("HTTPS dev certificate export failed for an unknown reason"); } }Note: The AppHost project must reference all the services used in the solution.
Keycloak client configurationSee the razorpagepar.json file in the git repository. This is a Keycloak export of the whole client. This can be imported and updated.
The client is configured to use PAR.
ASP.NET Core OpenID Connect client using OAuth PARThe client application uses the standard OpenID Connect client and requires OAuth PAR for authentication. This is a new feature in .NET 9. The repo has a Razor Page OpenID Connect example as well as an MVC client sample. This would be the same for a Blazor application.
services.AddAuthentication(options => { options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme; options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme; }) .AddCookie() .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options => { options.Authority = authConfiguration["StsServerIdentityUrl"]; options.ClientSecret = authConfiguration["ClientSecret"]; options.ClientId = authConfiguration["Audience"]; options.ResponseType = "code"; options.Scope.Clear(); options.Scope.Add("openid"); options.Scope.Add("profile"); options.Scope.Add("email"); options.Scope.Add("offline_access"); options.ClaimActions.Remove("amr"); options.ClaimActions.MapJsonKey("website", "website"); options.GetClaimsFromUserInfoEndpoint = true; options.SaveTokens = true; options.PushedAuthorizationBehavior = PushedAuthorizationBehavior.Require; options.TokenValidationParameters = new TokenValidationParameters { NameClaimType = JwtClaimTypes.Name, RoleClaimType = JwtClaimTypes.Role, }; }); Notes.NET Aspire looks great and is easy to use in development. I am only learning this and must learn the details now. I have some issues using the containers and HTTPS and I don’t understand how the configuration works. I also don’t understand how this would work in production. Lots to learn.
Linkshttps://www.keycloak.org/server/features
https://github.com/NikiforovAll/keycloak-authorization-services-dotnet
https://openid.net/specs/openid-connect-backchannel-1_0.html
https://github.com/dotnet/aspire-samples/tree/main/samples
https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview
I write this post on the first anniversary of the death of my dear father Jerry Lee Hawver who grew up in an unstable, alcoholic household and through hard work and great people skills became a star athlete and executive with Proctor and Gamble. He and my mother raised my brother and me in the corporate suburbia of the New South of the 1970s and 80s, steadily climbing the ladder of the American dream. For my formative years that was my culture. The dynamic in our home was that I was the child who sought approval. My brother pushed the boundaries. Over time he and I have come together to meet in the center. We are both Hawver stock, and we “get ‘er done.”
In those early years of the feral-child 1970s, us kids had the run of the neighborhood yards and wood lots. I’ve spoken of hanging out by the creek lined with quartz-laden rip-rap looking for crawdads and swinging from giant vines and making “forts.” My best friend’s name was Andrea Hopkins. She and her brother Brandon lived in a house that was kitty-cross from our backyard. Her father had a British accent and they had a few antiques, which I thought made them rather a glamorous family. We would often spend the afternoon exploring, riding our Big Wheels around, playing freeze tag, and putting on plays in the garage. Sometimes Andi and I would get crafty and make little coil pots out of the red Kentucky clay and fill them with potions made of petals from her mom’s flower beds along her driveway. Of course, there were marigolds in the mix. In some ways my new “cottage” rancher is like a return to this liminal space of childhood imaginative possibility.
I realize now that early success is not necessarily a blessing. I did well in school after a disastrous second-grade year with a body-building teacher, Mrs. Weigand, in an open classroom school I had to take three buses to get to (desegregation). I have a strange memory of being shown a graphic movie of the horrors of smallpox in a tiny room with carpeted levels. That made no sense to me, but I later found out that the Flexners (of the Carnegie / Rockefeller Flexner Report) were from Louisville, so maybe we were part of some public health behavioral experiment.
I recovered the following year at the much more traditional Wilder Elementary School where we had a fabulous librarian who would take small groups of kids around the shelves and do short book pitches and place books into the hands of kids she knew would love them. It’s funny that there were quite a few paranormal kids’ books in the 70s – John Bellairs and Zilpha Keatley Snyder. The one book that really stayed with me was Margaret Anderson’s “To Nowhere And Back,” a time-travel adventure where a girl passes through an English hedge into another century to make friends with a poor servant girl on the other side.
I was never in the popular crowd, but I always had a circle of friends. Without realizing it, my identity became wrapped up in my idea of performance, of being a “good” person based on what society said good was. My life unfolded with relative ease, meeting my husband in college, raising a small family, doing my part to make my neighborhood and city a better place, so I thought. But things started to change when my child was in middle school as I began to realize society was being broken on purpose, and the social systems I’d built my identity around began to give way one by one. Eisenhower’s chosen head for ARPA was the President of Procter and Gamble Neil McElroy? Oh, and they ran contracts building nuclear missiles in the panhandle of Texas (Pantex)? You don’t say? Well that sure put Mr. Whipple and Mrs. Olson in a new light in a new light for me. If one’s identity is wrapped up in societal expectations and those start falling away, how does the center hold?
What does it mean to be a “good person” within the expectations of a profoundly flawed system? Here my parents had worked so hard to walk us up the mountain of success, who was I to pull off on a wayside and kick rocks? I was the good girl, wasn’t I? When my marriage fell apart and my child became estranged from me, the silver lining was the opportunity to begin to shed the “good girl” archetype. Not that I had become a “bad girl,” it’s just that I recognized such labels were insufficient and did not do justice to the richly evolving tapestry of my life. Our lives have dramatic arcs. I know it sounds trite, but if there are no bumps in the road, you can’t truly appreciate the parts when the ride is smooth and clear.
Last year we were trying to sort out the memorial service for my father. At the time I had been applying to all sorts of jobs with the idea that I would stay in Philadelphia and try to hold onto our family home that was almost paid off. In my perceived role of the “good mother,” I thought at some point my young adult child might return from the west coast and need a place to land. Life laughed at me and had the literal soles fall off my shoes on the walk to one interview at Wharton.
During the two-day drive from Philadelphia to North Carolina to be with my father in hospice I cried a lot. I thought about his strength, both physical and strength of character. He was a big man, and I was small, but we fit together, a chip off the old block. I was listening to a podcast this week on our relationship to death and ancestors and Perdita Finn said that sometimes when people go, it is because they can help the ones they love more from the other side. I truly believe my father has been with me every step of the way since that morning when I held his swollen big hands and I read to him from L’Engle’s “An Acceptable Time” and bathed his forehead in kisses, my cheeks wet, and he crossed over.
After a hurtful email exchange with my child and husband about the memorial service, it was clear that I was being removed from their lives with strong intention and there was nothing I could do about it. I am a planner. Hawvers like to be in control. We try to fix things. All of those things were off the table. It was up to me to build a new story outside of all the stories that had defined my life since I met my husband during our study abroad program in Venice in 1989. I’d never been alone, and now I was going to figure out how to be me, for me, not the good daughter, the model student, the diligent mother and wife. I told my husband I was going to leave Philadelphia and get a cozy cottage where I could garden and a kayak. I spoke it outloud, a reasonable wish, with conviction and with a few days to spare I got it all squared away.
It’s taken me a year to come around to this way of thinking, because there was a lot of pain to acknowledge and process. But now I realize I was being given space to unfold in a new way. I had a choice. I could hold onto the baggage I was carrying and imagine myself as a failure across many categories. The thing is, I actually like myself and I don’t feel shameful about the course I’ve taken. Regrets? Yes, of course I have some. Have I made missteps? Certainly. But not out of malice, out of lack of information, insight, and wisdom. I deeply miss the people I have lost, but I can learn to be, to become, outside of them, outside their stories and expectations, and that’s ok, too. Perhaps some day they will have grown in their own stories enough and there will be enough space to hold all of who we have become in some new kind of family. Or maybe that won’t happen in this version of the story. I have to learn to accept what comes with an open heart.
Before he left this Earth, my father navigated Alzheimer’s for about six years with numerous falls, escapades, ditched cars, eating light bulbs, jumping back yard fences, and taking apart cable boxes at his nursing home. As I wade into a broader understanding of consciousness, I open up space in my heart for what was really happening with my dad and his glitching interface with what we understand as reality.
Over the course of my research this past year I found a very interesting memoir written by Elmer Green of the former Menninger Clinic in Topeka, KS where my dad grew up. It’s called “The Ozawkie Book of the Dead: Alzhiemers Isn’t What We Think It Is” (PDF) and it is the story of how he cared for his wife Alyce, his research partner in biofeedback, during her journey through that illness. It involves the nature of the soul and interpenetrating realms and the process of learning and unlearning how to be in this space.
I think I’m going to read it again, because as terribly traumatic as it was to see my strong, capable father engage with the world in ways that were so extreme and problematic and inscrutable, having a year’s perspective and new awareness of proto-consciousness and the unified field and emergent complexity, I can sit with the hard parts of the process and realize that maybe what my father’s struggles were doing was giving me a window to look through for the answers I seek.
I was under the weather for about a week. It started with a vicious migraine and then some shooting nerve pain and gradually resolved into a typical cold. Let me tell you, it’s not fun to be feverish in Arkansas in August. Most of the month the daytime highs have lingered in the upper 90s. By the end of the week I was regrouping, less foggy headed, and I decided to take a hike to Balanced Rock and make a heart to honor Jerry Hawver’s life. The trail is on the other side of town. It’s only a three-mile walk, and the idea of balance seemed appropriate. A friend had sent a delightful care package with a hummingbird card and dried herbs from her garden in Arizona. I clipped the hummingbird and took out a selection of peppers to place in my wicker basket along with a heart trimmed from the collar of one of his shirts. It still smells of his aftershave.
I drove past the bath houses downtown then parked my Subaru on the gravel pull out along Cedar Glades Road, quickly crossed the blind curve, and started the climb. The online reviews said that the first half mile took some effort. It was steep, sure, but the challenge was walking on loose scree. For years I never gave much thought to being particularly careful while I was out and about, because there were always at least two of us. Now, it’s just me. So I have to be watchful about taking a misstep and turning an ankle. I took things slowly and on my way up I picked up a chunk of Novaculite that seemed to have been worked into a hand sized cutting tool or scraper. Idly I ran my fingers over it as I pressed on through the dry, dusty woods.
Where it leveled out, I paused for a moment and a wild red and black bug caught my eye. When I looked it up later, it turned out to be a velvet ant, a parasitic wasp species where the females are wingless. A minute later, I saw her partner flying down the trail. Fortunately I had the good sense to keep my distance, as I later found out their sting is many times worse than a hornet’s and they are commonly called “cow killers.”
An interesting synchronicity is that in one of my early blog posts I warned about hybrid “personalized” learning being like a cicada killer. A cicada killer is a large wasp that frequented the garden where I worked. Its reproductive cycle involved capturing their chosen prey on the wing, paralyzing it, dragging it into an underground burrow, and then laying eggs on the body where the larvae would eat their way out. It was rather grotesque imagery, but it served to make a point. Well, turnabout is fair play, and evidently these velvet ants parasitize the pupae of cicada killers. Hmm.
Despite the dryness you could still find a few ferns, and where the side trail to Balanced Rock branched off there were some wild plums dropping onto the path. I gathered a few things en route to supplement what I’d been gifted and brought from home. Not long after I arrived at the outcropping – more loose scree, more careful steps. I managed to get far enough down the slope to fully appreciate the balance embodied by these two impressive boulders.
Of course right as I got there, a couple arrived panting, and so we gave each other space. I pulled over to a little overlook in the shade and pulled out my basket of offerings. The arrangement of items in this particular heart is both loose and disordered while being vibrant and tactile. There are cracked nuts full of labyrinths, fractal turkey tail fungus, the first fall leaf offerings, and peppery spice.
I think that would suit my dad just fine. By the end everything was coming apart, which is a difficult thing for Hawvers. Have I said before that Hawvers like to be in control and have a plan? Well, we do. And the last years of Jerry’s life were anything but that. At the time there was a heaviness about it, this uncontrollable fraying and sense of loss that would not be diverted, but a year later, perhaps I can reimagine it as a cocoon tearing apart the restricting expectations my dad must have felt to be a provider and a fixer and the rock for all of those years. To transition to a new form meant things were going to fall apart. How else is renewal possible? In the looseness of MIA control systems there is the unrealized potential of a new blueprint and within the chaos, a kernel of what’s next.
I sat on that bit of bluff, and thanked my dad for all the things he provided for us. I can feel him giving me a big bear hug and me offering in return a big back scratch. I told him that I missed him deeply and thought of him every day. I also said that I knew he had been walking beside me, but that I was going to be ok, that if he felt held back in any way by me, he could go. I can just picture him high-fiving people all over heaven. I prayed that the creator would help me have the clarity to see the signs and find my purpose as I unfold into this new life. Then I packed up and headed back down the trail.
The walk back was uneventful with one exception. Near the far end I paused before an unusual artifact, a rusted out wash basin hanging from a knot in a tree at eye level, almost where a shaving mirror would be. It had been galvanized, but the rust had gotten to it and the entire bottom was eroded away. I lifted it off the knot to examine it more closely. It wasn’t large, a bit wider than a typical dinner plate and about four inches deep with a narrow rim punctured in one spot where you could hang it from a nail.
Now it wasn’t a totally bizarre addition to the landscape. You might find something like that tossed in a ravine out behind an old farmhouse. But there was no other debris around, no trash. I found it right along a trail on land that had been a national park for almost a century. It wasn’t tucked away in the brush; it was right in front of my face. It wasn’t a plastic bottle or even a face mask carelessly tossed trail side. No, this was a message for me, this wash basin, that if it weren’t for the rusted out bottom, would be right at home as part of a Cracker Barrel wall display.
Ok, so I had asked for the wisdom to notice the signs, but then what was I to make of this out-of-place, out-of-time basin? One story you might tell is one of obsolescence. When my life first fell apart during the lockdowns, that was the story I initially jumped into.
Who am I now that I am no longer my child’s mother, my husband’s wife, my mother’s accomplished daughter? I was a basin whose design was to hold things for others. Basins don’t function alone, they contain things. For decades my identity had contained many things, big and small. From carpool schedules and family vacation plans, to dinner menus and school registration procedures, civic campaigns, etc. etc. My existence was to be a helpful presence. Then somehow I picked up some rust. Perhaps at that school report card meeting in the summer of 2013. Rust wore away at the old functionality of the basin. Interestingly rust interferes with magnetism in metal. So as the basin’s rust surface area increased, it became less polarized. I imagine that might represent a shift towards non-dualism.
Was I meant to believe I was a failed basin? No, of course not. Because I believe in a benevolent creator that is loving and wants what is best for us. So the story I am choosing to tell myself about the basin is one that Alison Hawver might have imagined in third grade in Mrs. Nevius’s cardboard geodesic dome reading nook sitting on a carpet square. I am going to imagine that like the ugly duckling becoming a graceful swan, I have been a capable and trustworthy basin for many years, but slowly grew into a magical portal of imaginative potential, which given the state of the world right now, is a pretty awesome skill to have on offer. The twenty-first century needs some wiggle room for new possibilities to emerge, some fluidity to flex the boundaries of spacetime.
I’ve traded in the closed, hygienic system of wash basin culture for the jagged, unexpected potential of a back-to-nature, quantum vacuum, anti-mirror portal into proto-consciousness where we can dip in for messages from worlds with a better view than ours. A portal like the cottage hedge in the book “To Nowhere and Back” or the star watching rock from L’Engle’s Time Quintet that I read from at my father’s death bed.
Well, I’ve written all day, and the heat has finally broken with a gentle rain and a cool breeze here on the crystalline flank of Spa City. I hope you haven’t found this too self-indulgent or maudlin. There are lots of big feelings to navigate these days for all of us, and sometimes it’s useful to remember back to the times when all we had to do was make petal potions with our backyard buddies. If you quantum foam some interesting artifacts into your life, I’d love to hear about them and the story goes with that. Thanks for hanging out with me on this Labor Day weekend 2024. A toast to Jerry Hawver. I wouldn’t be who I am without him.
Dear readers,
For those who are still following along, you’ve probably noticed it’s been quiet on the blog this summer. I did a short update on my channel a few weeks ago letting folks know I found a cozy home, a 1970s two-bedroom brick rancher on about a quarter acre on the outskirts of Hot Springs, AR. It is a sturdy little house with a sunny yard and quiet neighbors, mostly retirees who are kind about looking out for you without intruding and generously extend weekly invitations (in season) to share enormous slabs of delicious ice cold water melon. It’s nothing fancy, but it after a month I can say it suits me perfectly and I am grateful for the blessings of having a safe place to embark on this next phase of life.
My snail mail address is 123 Marigold Circle Hot Springs, AR 71913. I’ve already received a few letters and care packages, which are much appreciated. I’m loving the slower pace of things in my chosen exile. Maybe I can have a go setting up some old school pen pal correspondences if you care to drop me a line.
Marigold Circle is a loop at the end of Hobson Road that is tucked into the base of one of the ridges underlaid by that famous quartz crystal that comprise Hot Springs National Park, the oldest federally protected land in the country (1830s compliments of Andrew Jackson). This development of about thirty homes, locally known as “Little Chicago” is situated where West Mountain begins to transition into Music Mountain. For years Hot Springs was known as a mobsters’ hang out, some with Chicago ties. Oh, and the circle is a square (not a cul-de-sac), and I’m in the middle of the doughnut (aka the quantum vacuum, wink, wink).
I’m loving the breathing room provided by the small open lawn around my house that gives me space to take in the blue Arkansas skies, often punctuated by dramatic towering clouds. There is considerably less geoengineering here than in Philadelphia. It does happen, but much less. The forest that embraces our little development is a mix of pine and hardwoods, so I should have nice vistas at all times of the year. It’s not an off-the-grid, bug-out place; but it is quiet, there are good people here, and lots of places to hike and get out on the water. I bought an inflatable stand up paddle board that can convert into a basic kayak, which will be easier for me to transport. Now that I’ve got my life vest, I’m going to head out AFTER the Labor Day boating hoopla and soak up a bit of the calm of the shoreline of Lake Ouachita and the 1.8 million acres of national forest surrounding it. Water is the theme of the year.
My yard doesn’t have trees, but my neighbors do – a sweetgum and sycamore on the south, a maple to the north. It’s been a very hot month with daytime temps rarely getting out of the upper nineties and almost no rain. The poor gum tree, in particular, seems quite stressed. Having lived in a row house for the past thirty years, I didn’t own a rake, just a broom for the front walk, so I grabbed a solid wooden-handled Craftsman at Lowe’s this week. Even though I got a blower with my lawn mower purchase, I do hate the sound of them. For me Hot Springs is a great chance to slow down and get back to the basics. I slowly filled half a contractor bag with leaves yesterday and look forward to getting a better composting situation set up soon.
There’s a towering “axis mundi” oak across the street. It has been trimmed back by the power company, so it appears almost split in half, its impressive fractal branching pattern is on full display, which is right in line with my recent musings on complexity and fractals. This particular specimen is over sixty-feet high. Given the severe weather you can get here, I recently took out a measuring tape to see how much distance there was between its base and my eaves. Hopefully, if it ever falls, only the top ten feet will hit my house, enough to mess things up, but not totally destroy the house.
The one poor choice I’ve made so far (that I know of) is that I initially bought a terribly cheap, uncomfortable twin mattress. I don’t have sleep issues or chronic pain and don’t consider myself a mattress snob. Our old queen mattress that we had for over twenty years held up like a champ. It was just a basic Serta or something, but of course they don’t make them like they used to. I ended up buying a replacement with natural materials that should be arriving next week. I’ll put the old one in a mattress bag in the basement for tornado season. It’s the perfect use for a sad excuse of a sleeping surface, perhaps one day padded protection from falling timbers. If that badly-pruned oak ends up taking a nosedive through my living room, it could end up being the hero of my story.
My house is on a slightly fall-away lot, which is why I have a basement. Happily, it is still level enough to garden once I get a fence and raised beds set up to deter the many deer and bunnies. All the houses are modest one-stories, and so my little back sun porch has the feeling of a tree house. The neighbors feed the birds (and the deer). Most days I feel I’m simply a visitor in their avian kingdom. Marigold Circle is an ideal woodland edge habitat. There is a raucous group of jays, daredevil mocking birds and thrashers, lots of flickers, turtle doves, cardinals, house finches, and Carolina wrens. Some evenings or early mornings I get lucky to catch a glimpse of a hummingbird probing the magenta trumpets of the Four O’Clocks in my front dooryard. There are lots of dragonflies and butterflies and wasps, too. The wasps pollinate the big turkey fig out back that has been offering me a handful of sweet fruits every morning since I moved in.
I mentioned in my video that I’m weighing the extent to which it makes sense for me to continue to share my research moving forward. I’ve pretty much unplugged. I’m not on social media outside of the interactions I have on my channel and blog. I don’t have a television or radio. I have a portable “suitcase” record player and am gradually collecting bluegrass and folk on vinyl. I don’t check the news online. What information filters in comes to me through my small circle of friends. I’m less interested in blogging as a contact / “gotcha” sport, as tempting as that is with all the things we’ve been talking about now rolling out including Crypto Bobby, Trump’s new sidekick.
You might think that is irresponsible of me to check out, but I have come to the realization that I am not going to change the world through social activism or civic engagement. Trying to convince people to pay attention to the story I have attempted to surface doesn’t work. The learning curve is too steep and the feed does not operate in my favor. That said, maybe there is another way to tweak the seemingly chaotic but perhaps not entirely complex labyrinth of what we understand to be reality. Maybe it is my task for now to try and put my intuition and intellect into puzzling out some cheat codes and rather than throwing a wrench in the gears to stop the progress of so many problematic programs, learn to play living systems instrumentally with heart by adjusting initial conditions and tuning constraints. It may be a fool’s errand, but for now I turn inward – not giving up, but handing over control to the field of imaginative possibility as I try and train my eyes to see the clues and piece them together. Being quiet and reading and sewing for the time being is enough. I have put in my time trying to be loud. In the end it was unsatisfying, and even destructive to my wellbeing.
The idea of responding to breaking developments isn’t satisfying. Neither is critiquing personalities or asserting the superiority of my lens. Mostly, I want to continue to follow the intellectual meanderings that feed my soul. Neal Stephenson’s novel, “Anathem” revolves around a society of monastic thinkers who remove themselves from the world, so they can focus on weightier contemplations. I can see the logic in that. Stepping outside the noise has brought me a measure of peace and calm. Walking around barefoot, something I couldn’t do in Philadelphia, has felt great and is very grounding, too!
The physics of digital information is different. I think it comes so fast and furious and disconnected from our actual lived experience that it overwhelms our sense-making instincts both individually and collectively and ends up leaving us anxious and adrift. I feel like I know enough for the moment about the basics of web3, the mechanics of social impact policy and digital governance, and the Internet of Bio-Nanothings. Now I want to better understand the realm that we are in. Not in an esoteric, occult way that appeals to many; more like a Claude Shannon, information theory way. I think energy flow and information is the key to traversing the choice architecture of inner space. The more I discover, the less certain I am about anything. I do think there is merit in putting my thoughts down in writing with a date and time attached, even though I’m less and less invested in the concept of linear time. That said, the postings here will probably continue to be sparse.
The topics I currently find to be most compelling are quantum biology, proto-consciousness, complex non-linear dynamical systems, morphic fields, parallel processing, distributed collective intelligence, the observer effect, narrative structures, spatial computing, responsive environments, digital artifacts and deep learning. I am dipping into a lot of information flows along with friends and I’m not exactly sure what to say about my thought process yet. In addition to Karl Friston’s Free Energy Principle and Chris Fields’ Active Inference work, I am also very interested in the ideas of the late Jungian psychiatrist, dream analyst, and quantum consciousness theorist Gordon Globus. I am wrapping up his 2009 work “The Postmodern Mind” now.
I have a lot of ideas about non-traditional “computing” interfaces and humans-in-the-loop that have less to do with nanotechnology than with Qi, psi and plasma and etheric realms. As the head of standards for IEEE once said – it’s really all metaphysics!
Minimizing surprise and making solid predictions is an evolutionary imperative, according to Karl Friston’s free energy principle. We hold a model of how the world works in our heads, so that we can use it to make good choices navigating uncertainty. Many of those choices are fluid and socially normed. Some people are more comfortable changing their model than others. Some people deal with surprise by looking away. Then there are the odd balls like me who keep hoovering up information and trying to find patterns in it so that we can conjecture about stories that could make the whole messy pile hold together for a time at least, because there are always more surprises awaiting us.
It seems in my case, I became too much of a surprise for my beloved family, and the perceived cost of changing their models to accommodate my admittedly fringe world view was too high. I can’t say they were wrong. I will simply say I didn’t realize with each small step, new nugget of information, and adjustment that I was being removed from their world model. I simply did not appreciate how profound the cost would ultimately be.
I am still working to make sense of what happened to me in a way that is non-judgemental and centers compassion for all involved. I do believe understanding the power of the heart field is vital to navigating the turbulence of the world – not just in a sweet, saccharine sort of way but in a quantum processing sort of way that connects us straight to the divine nature of the cosmos.
I have another blog post I plan to work on tomorrow in honor of the year anniversary of my father’s passing. I’ve been under the weather for about a week, but it seemed to be resolving so a few days ago I took a short out and back hike to Balanced Rock on the other side of downtown. On the return leg I encountered an unusual artifact, which felt like it had some symbolic significance. My next writing will be a reflection on that.
I’ll close with an entertaining and informative talk my friend Sean posted this week about artifacts inspired by a discussion I’d posted with Lynn on Open Education Resources. Sean’s talk features a read aloud from a Mrs. Frizzle and the Magic School Bus book sponsored by the Department of Energy. It’s a good format, be sure to check it out.
All the best,
Alison McDowell
123 Marigold Circle
Hot Springs, AR 71913
PS: I just learned that marigolds go by the Latin name Tagetes erecta, which comes from Tages the Etruscan deity of divination. West is the direction of the underworld in Greek mythology, though I continue to imagine these other realms as being more about signals and information, which means it’s interesting that the ridge above me is West Mountain and Music Mountain.
I just did a massive spring cleaning of one of my servers, trying to clean up what has become quite the mess of clutter. For every website on the server, I either:
Documented what it is, who is using it, and what version of language and framework it uses Archived it as static HTML flat files Moved the source code from GitHub to a private git server Deleted the filesIt feels good to get rid of old code, and to turn previously dynamic sites (with all of the risk they come with) into plain HTML.
This is also making me seriously reconsider the value of spinning up any new projects. Several of these are now 10 years old, still churning along fine, but difficult to do any maintenance on because of versions and dependencies. For example:
indieauth.com - this has been on the chopping block for years, but I haven't managed to build a replacement yet, and is still used by a lot of people webmention.io - this is a pretty popular service, and I don't want to shut it down, but there's a lot of problems with how it's currently built and no easy way to make changes switchboard.p3k.io - this is a public WebSub (PubSubHubbub) hub, like Superfeedr, and has weirdly gained a lot of popularity in the podcast feed space in the last few yearsOne that I'm particularly happy with, despite it being an ugly pile of PHP, is oauth.net. I inherited this site in 2012, and it hasn't needed any framework upgrades since it's just using PHP templates. My ham radio website w7apk.com is similarly a small amount of templated PHP, and it is low stress to maintain, and actually fun to quickly jot some notes down when I want. I like not having to go through the whole ceremony of setting up a dev environment, installing dependencies, upgrading things to the latest version, checking for backwards incompatible changes, git commit, deploy, etc. I can just sftp some changes up to the server and they're live.
Some questions for myself for the future, before starting a new project:
Could this actually just be a tag page on my website, like #100DaysOfMusic or #BikeTheEclipse? If it really needs to be a new project, then: Can I create it in PHP without using any frameworks or libraries? Plain PHP ages far better than pulling in any dependencies which inevitably stop working with a version 2-3 EOL cycles back, so every library brought in means signing up for annual maintenance of the whole project. Frameworks can save time in the short term, but have a huge cost in the long term. Is it possible to avoid using a database? Databases aren't inherently bad, but using one does make the project slightly more fragile, since it requires plans for migrations and backups, and If a database is required, is it possible to create it in a way that does not result in ever-growing storage needs? Is this going to store data or be a service that other people are going to use? If so, plan on a registration form so that I have a way to contact people eventually when I need to change it or shut it down. If I've got this far with the questions, am I really ready to commit to supporting this code base for the next 10 years?One project I've been committed to maintaining and doing regular (ok fine, "semi-regular") updates for is Meetable, the open source events website that I run on a few domains:
events.indieweb.org events.oauth.net events.scim.cloudI started this project in October 2019, excited for all the IndieWebCamps we were going to run in 2020. Somehow that is already 5 years ago now. Well that didn't exactly pan out, but I did quickly pivot it to add a bunch of features that are helpful for virtual events, so it worked out ok in the end. We've continued to use it for posting IndieWeb events, and I also run an instance for two IETF working groups. I'd love to see more instances pop up, I've only encountered one or two other ones in the wild. I even spent a significant amount of time on the onboarding flow so that it's relatively easy to install and configure. I even added passkeys for the admin login so you don't need any external dependencies on auth providers. It's a cool project if I may say so myself.
Anyway, this is not a particularly well thought out blog post, I just wanted to get my thoughts down after spending all day combing through the filesystem of my web server and uncovering a lot of ancient history.
Programming note: instead of our usual The Pulse on Thursday, today we peek inside a book that I wrote the foreword for. Our regular schedule — and The Pulse — returns next week.
Addy Osmani is a software engineer and engineering leader. He’s worked at Google for 12 years, and is currently the Head of Chrome Developer Experience. Addy regularly shares insights on software engineering and engineering leadership, and is the author of several software engineering books. He also writes the newsletter .
When I was visiting the Bay Area, we met up with Addy, who gave a tour of the Chrome offices:
With Addy, in the Google Chrome team’s lobby, in Mountain ViewAs we talked, he shared how he’s been working on a book about engineering leadership – collecting a decade of lessons learned in motivating and leading engineering teams. The lessons span Addy’s earlier days – when he was a software engineer, and was mentoring fellow devs – through working as a tech lead manager (a specialist role present in larger tech companies like Google) – all the way to leading larger engineering teams. I was intrigued, and asked for an early look. In the end, not only did I read the book ahead of release, but I found it such a neat mix of “theory” and “practice” that I volunteered to write the foreword.
The book is now out, and I asked Addy if he’d be open to sharing two relevant parts with all newsletter readers. Addy said yes, and so in this issue we take a look inside the book, covering:
Context on the book. Why write this book, how long it took, and Addy’s favorite part of it.
What Makes a Software Engineering Team Effective? No company invested more research in this area than Google. A brief summary of Project Aristotle and Project Oxygen, and a collection of other useful team dynamics research.
Leadership roles: tech lead, engineering manager, and tech lead manager. A look at how these three roles overlap, and also differ. The tech lead manager (TLM) is unique to Google, and a few similar companies, and is the most unusual of the three.
My usual disclaimer: as with all my recommendations, I was not paid to recommend this book, and none of the links are affiliate ones. See my ethics statement for more details.
1. Context on the bookHow did the idea for writing this book come around? This is what Addy told me:
“The idea for the book started percolating a few years back. I'd been in the trenches of engineering leadership at Google, and I was seeing patterns – ICs, leaders and managers all cared about being effective, but there wasn't a well defined resource on this topic. I would email folks guidance whenever they would ask, but there was a real disconnect between the academic world of management and the gritty reality of leading high-performing engineering teams.
People were hungry for practical guidance, not just theoretical frameworks. That's when I realized there was a need for a book that could bridge that gap.
So I started working on my notes that would eventually turn into this book.”
Writing a book takes a long time, and I was curious how much effort this one took. It was 1.5 years to write – plus the many years of experience to have something worthwhile to pen down. From Addy:
“Writing the book was a longer haul than I expected. Writing a book is a bit like building a complex software system - it requires careful planning, execution, and constant iteration.
While the actual writing took about a year and a half, the foundation was years in the making. It was essential to blend my personal experiences with rigorous research. I wanted to ensure the book wasn't just a collection of anecdotes but a practical guide backed by data and insights. I think as a team (O'Reilly, our editors and tech reviewers as well) we were able to land on something in the end that we all felt proud of.”
The book has seven chapters, and I was curious as to what Addy’s favorite part is, if there’s any. Addy said:
“The 3 E's framework (enable, empower, expand) is undoubtedly the cornerstone of the book. It's something I've been refining over years of leading teams. I'm particularly proud of how it provides a practical approach to building high-performing engineering organizations.
What excites me most is how the model can be adapted to different team sizes and company cultures. It's not a one-size-fits-all solution, but a flexible framework that can guide leaders at various stages of their career. I'm eager to see how others apply it and share their experiences.”
With this added context, let’s dive into two chapters of the book.
The below sections are from Leading Effective Engineering Teams, by Addy Osmani. Copyright © 2024 Addy Osmani. Published by O'Reilly Media, Inc. Used with permission.
2. What Makes a Software Engineering Team Effective?The below is from the beginning of Chapter 1 in the book.
Some teams seem to operate like well-oiled machines, churning out successes. Communication flows seamlessly, they meet deadlines with a smile, and they tackle challenges head-on. Conversely, other teams struggle to reach every milestone. Communication is chaotic, and meeting deadlines is a challenge. What makes the successful teams effective? It’s usually a mix of things: clear plans, honest talk, a healthy dose of trust, and a shared belief in what they’re doing. Some teams already have the rhythm and the steps down pat, while others are still figuring things out. But the good news is that everyone can learn the steps. Even the most stumbling crew can find its rhythm with a little practice.
This rhythm manifests itself in software engineering teams as their ability to produce useful products or product features by writing code, testing it, and releasing it to the world. Teams that do this regularly are said to be effective. So, to build great software, we must first build effective engineering teams.
Throughout my 25+ years of experience leading engineering teams at Google and other tech companies, I’ve seen firsthand how team dynamics can make or break a project. Building effective teams is not just about assembling the right technical skills; it’s about fostering a culture of collaboration, trust, and shared purpose. In this chapter, I’ll share some of the key lessons I’ve learned about what makes engineering teams successful, drawing on both research and my own experience in the trenches.
What makes an engineering team effective hinges on the key thing that distinguishes teams from groups. On the one hand, a group is a collection of individuals who coordinate their efforts. On the other hand, a team is a group that is bound by shared responsibilities and goals. Their members work together and share mutual accountability to solve problems and achieve common goals. When teams plan their work, review progress, or make decisions, they consider the skills and availability of all the members and not just those of one individual. This shared goal is what drives an effective team.
I have had the opportunity to observe or be a part of such teams at Google. These teams are passionate about achieving their goals. They find brainstorming sessions fun rather than stressful. Team members may write and test code on their respective machines, but they are collectively tuned in to a unified vision of what the code should achieve. There have been times when they had to resolve some difficult issues, but a culture of collaboration, innovation, and mutual respect helped to see them through such times.
Leaders are an important part of this picture. As a software engineering leader who wishes to make your team effective, you serve as an anchor that connects individual team members to the shared responsibilities and goals of the team. You provide the vision, direction, guidance, and environmental framework necessary to form this connection.
Although it’s possible to have a team without a leader, the team will go much further with the support of a good leader—and that’s where you come in!
Building an effective software engineering team takes work. Many factors can influence the success of a software engineering team, such as team composition, communication, leadership, and work processes. This chapter will explore what traits make teams effective and how to build them into your team. These traits will be things you can look for when hiring, but they’re also traits you can nurture in your existing team.
Research on What Makes Teams EffectiveFirst, let’s examine what makes teams effective. To do so, let us look at some of the extensive research that has already been done on this topic.
Project AristotleGoogle conducted one of the best-known studies on effective software engineering teams, known as Project Aristotle. The project aimed to identify the factors that make some teams more successful than others. The study was based on the premise that the composition of a team was not the most critical factor in determining success but rather how team members interacted with each other.
Note: Before Project Aristotle, there was Project Oxygen, which looked into what traits make for a good manager. Some of the insights in this chapter were informed by the results of Project Oxygen, which I’ll talk about in detail in Chapter 4.
To determine what makes teams effective, the researchers first had to define what effectiveness means and how to measure it. They noticed that different roles had different perspectives on effectiveness. In general, whereas executives were interested in results such as sales numbers or product launches, team members thought that team culture was the key to team effectiveness. The team leaders indicated that ownership, vision, and goals were the most important measures.
Eventually, the researchers decided to study certain qualitative and quantitative factors that might impact team effectiveness, such as the following:
Team dynamics. Demographics, conflict resolution, goal setting, psychological safety
Personality traits. Extraversion, conscientiousness
Skill sets. Programming skills, client management
Researchers conducted interviews and reviewed existing survey data for 180 Google teams. They used this data to run 35 different statistical models and understand which of the many inputs collected impacted team effectiveness.
Project Aristotle identified five key dynamics that contribute to the success of software engineering teams (see Figure 1-1). These are listed next in the order of their importance:
Psychological safety
This was the most important factor identified by the researchers. It refers to the extent to which team members feel comfortable expressing their opinions and ideas without fear of retribution or criticism. Teams that have high levels of psychological safety tend to be more innovative and take more risks, which can lead to better outcomes. The researchers found that when teams feel safe, they:
Are less likely to leave the company
Are more likely to utilize the diverse ideas discussed by the team
Bring in more revenue and beat their sales targets
Tend to be rated highly on effectiveness by their leadership
Dependability
This refers to the extent to which team members can rely on each other to complete their work and meet deadlines. Teams in which individuals trust each other to be dependable are more likely to be efficient and effective in their work.
Structure and clarity
These are conditions under which team members clearly understand the project’s goals and their own individual roles and responsibilities. Team members who clearly understand what is expected of them tend to be more productive and focused.
Meaning
This refers to the extent to which team members feel that their work is meaningful and has a purpose. Teams with a strong sense of purpose tend to be more motivated and engaged.
Impact
This refers to how team members believe their work is making a difference and impacting the organization or society. Teams with a strong sense of impact are more committed to their work and the project’s success.
Figure 1-1. Google’s Project Aristotle: The five dynamics of effective teamsWhile Project Aristotle’s research was conducted within Google, the identified factors influencing team effectiveness could hold some relevance for teams in other contexts. By focusing on these five factors, software engineering teams can create an environment conducive to collaboration, innovation, and success. As I’ll discuss in Chapter 4, a good manager can foster these dynamics in their teams.
The researchers also discovered that variables such as team composition (size and colocation) or individual attributes (extroverted nature, seniority, tenure, etc.) did not contribute significantly to team effectiveness at Google. While these variables did not significantly impact team effectiveness measurements at Google, that doesn’t mean they’re unimportant, as indicated in the following section.
Other ResearchWhile Project Aristotle is perhaps the best-known study on effective software engineering teams, many other studies have explored factors such as team composition, communication, leadership, and work processes. Here are a few key findings from some of these studies:
Smaller teams are better.
Although Project Aristotle did not recognize team size as relevant to team effectiveness, other studies have shown that smaller teams work better. As a team gets bigger, the number of links that need to be managed among members increases exponentially. Managing these multiple communication channels can be complicated. Many researchers have identified smaller teams containing less than 10 members as more likely to achieve success than larger teams.
Diversity can be beneficial.
It is sometimes suggested that team diversity may lead to communication and coordination problems. For example, a diverse team would usually consist of people from different family backgrounds. Those with young children are more likely to seek flexible work hours, leading to coordination challenges. However, others have found that diverse teams can be more innovative and effective. A study by Lu Hong and Scott Page of the University of Michigan found that groups of randomly selected (likely diverse) high-ability problem solvers can outperform groups comprising the best problem solvers. However, it’s important to note that diversity alone is not enough. Teams must also create an inclusive and respectful environment for all team members. For example, a team that is supportive of team members who need flexible work arrangements will be able to coordinate better than a team that is intolerant of members with such needs.
Clear communication is vital.
Effective communication is essential for effective teamwork. Studies have found that teams that communicate frequently and openly are more successful than those that do not. The idea of psychological safety is a shared belief among team members that they can freely express their thoughts, ideas, concerns, or even mistakes without fear of negative consequences or judgment. Its importance is backed up by the research from Project Aristotle. Clear communication also provides the glue to connect team members and establish structure and clarity within the team.
Leadership matters.
The leadership of a software engineering team can have a significant impact on its success. Google’s Project Oxygen showed that although teams could function without a leader, there is still a need for managers. It identified the essential traits that make for good managers and effective teams. I will talk about these traits in Chapter 4, but for now, it’s necessary to understand that there is a strong correlation between effective leadership and positive team outcomes.
Agility enables adaptability.
Agility is the ability to adapt quickly to changing circumstances. In software engineering, this means being able to pivot when requirements change or when unexpected issues arise. Agile teams are quick to adapt and can work swiftly and efficiently while maintaining high quality. A study by McKinsey & Company found that organizations that underwent successful agile transformations reported a significant improvement in efficiency, speed, customer satisfaction, innovation, and employee engagement, all of which are essential to effectiveness.
Colocation powers innovation.
The debate over whether colocation or remote work is better for software team effectiveness is ongoing, with both approaches having their own advantages and disadvantages. Multiple studies conducted at Harvard, Stanford, and others discuss the benefits of remote or hybrid work in terms of employee satisfaction and retention. However, other studies have shown that face-to-face interactions at the workplace, both planned and serendipitous, trigger the flow of knowledge, sharing of values, and exchange of ideas, which contribute to innovation.
While there may be trivial differences in the findings, we can build a theoretical picture of an ideal effective team based on the research findings discussed in this section. See Figure 1-2. By enabling psychological safety, clarity of structure and communication, dependability, meaningful work, and agility, software engineering teams can create an environment conducive to collaboration, innovation, and success.
You can now build on this understanding of dynamics and factors that influence the effectiveness of teams. The next things to consider are how the working environment can affect teams and how motivation can prime your team for success. As you go through the next sections, notice how the factors that affect teams pop up in various contexts.
3. Leadership Roles: TL, EM, TLMThe below is an excerpt from the middle of Chapter 7: Becoming an effective leader
Organizational structures in software engineering organizations differ widely depending on their culture and priorities. After a person has served as an engineer or senior engineer for a few years and gained the necessary expertise, there are typically two tracks open to them: technical or managerial. Each offers distinct leadership opportunities and requires individuals who can coach and guide their teams through challenges.
In this section, you will look at some typical roles across the industry and what they usually entail in terms of effective leadership. Note that these aren’t the only leadership roles in an organization.
Leadership roles in a team depend not only on the overall organizational structure but also on the size and complexity of the project. Larger teams could have one or many technical leads leading the development of different parts of a project. Additionally, such teams would have architects synchronize the efforts led by the technical leads and managers to plan and organize resources. You could also have a product manager who articulates what success looks like for a product and guides the team to make it a reality. Conversely, in small teams, these roles may be combined to have a manager with the technical expertise to lead the team.
Figure 7-2 shows how some of the different types of leadership roles may coexist in a software engineering team.
Figure 7-2. Relationships between various leadership roles in a software engineering teamLet’s take a closer look at some of these leadership roles.
Technical LeadA technical lead is a hands-on role where you provide technical guidance and direction to the engineering team. The designation itself may vary across organizations. It may be a formal title in some workplaces, while it exists more informally in others. In some organizations, the position may be identified as a “software architect,” while in others, it may be referred to by titles like “principal engineer” or “lead software engineer.”
Irrespective of the name, tech leads play a crucial role in architectural decisions, code reviews, and mentoring junior team members. Technical leads often bridge the gap between the development team and management, ensuring alignment between technical strategies and business goals. Some of the responsibilities of a technical lead include the following:
Guide technical design and architecture
Tech leads play a vital role in shaping the technical direction of the project by providing guidance on design and architecture. A tech lead must leverage their expertise to ensure that the chosen technical solutions align with the project’s goals and adhere to industry best practices.
Set coding standards and best practices
Tech leads should take the initiative to establish coding standards and best practices within the development team. The tech lead role involves defining and enforcing these guidelines to contribute to overall code quality, maintainability, and consistency.
Lead troubleshooting of complex bugs and issues
Someone in the tech lead role leads the investigation and resolution of intricate technical issues and bugs. Their deep understanding of the codebase empowers them to troubleshoot effectively, ensuring the stability and reliability of the software.
Make key technical decisions with engineering trade-offs
Tech leads are responsible for making critical technical decisions, carefully weighing engineering trade-offs to align with project objectives. They consider factors such as performance, scalability, and maintainability to ensure the overall success of the software.
Do hands-on coding alongside the team
Despite their leadership role, tech leads often find themselves actively engaging in hands-on coding alongside their team members. This approach helps them mentor other engineers while staying connected with the codebase.
Serve as a mentor for development skills
Tech leads also act as overall mentors, guiding team members to enhance their development skills. They lead by example to foster a culture of continuous learning and professional development within the team.
Ensure deliverables meet the quality bar
Tech leads are accountable for the quality of deliverables, ensuring that the software meets established standards and requirements. They conduct thorough reviews and quality assessments to guarantee that the end product aligns with the defined quality bar.
Depending on the size of the project, the scope of these responsibilities will vary—from overseeing a single development team to having cross-team responsibilities.
Engineering ManagerAn engineering manager typically oversees a team of software engineers, ensuring the successful delivery of projects. They are responsible for project planning, resource allocation, team productivity, performance, and career development, including that of the tech lead. This role often involves a mix of managerial tasks, such as performance evaluations and career development, along with technical oversight. In some companies, engineering managers may also be referred to as “development managers” or “technical managers.” To recap, an engineering manager’s key responsibilities include the following:
People management
Engineering managers should gear up to develop their skills in hiring, talent development, coaching, and mentoring. Engineering managers actively engage in the recruitment process, nurture their team members’ potential, provide guidance, and foster a culture of continuous learning within their team.
Manage processes
Engineering managers orchestrate critical processes such as sprint planning, retrospectives, and regular one-on-ones. They should ensure these processes are not just executed but tailored to their team’s needs, promoting collaboration, communication, and continuous improvement. They need to check that processes are not sidestepped.
Align team with organizational priorities
Engineering managers must ensure that their team is aligned with the broader organizational priorities. This involves effectively communicating context, goals, and expectations to team members while also shielding them from unnecessary distractions. By serving as a bridge between the team and the larger organization, the engineering manager helps team members focus on their work and deliver value.
Unblock resources
Engineering managers must actively work on unblocking resources needed for execution. They liaise with other departments, manage dependencies, and ensure that their team has the necessary tools, resources, and support to deliver on their commitments.
Technical oversight
While the engineering manager may not have any hands-on coding time, they should maintain their technical acumen. They engage in architecture discussions, ensuring technical decisions align with best practices and organizational goals. This technical oversight helps them guide their team to find sound technical solutions.
Stakeholder interaction
Engineering managers should engage with stakeholders, including having direct interactions with customers. They must understand project requirements, ensure proper communication channels, and act as a conduit between their team and external stakeholders. Engineering managers ensure that the team receives clear requirements from stakeholders.
Strategic work prioritization
Engineering managers must strategically prioritize work aligned with their team and company’s vision. This involves balancing project commitments with essential operational work, addressing technical debt, performing and maintenance in line with the organization’s strategy.
As you take on an engineering manager role, remember that you must broaden your responsibilities to include comprehensive people management, process leadership, and strategic alignment with organizational goals in addition to technical oversight. Unblocking your programmers is also an essential but slightly underrated aspect of managerial responsibilities.
Joel Spolsky, the cofounder of Stack Overflow and creator of Trello, once said, “Your first priority as the manager of a software team is building the development abstraction layer.”1 He further explains that if a developer is directly exposed to infrastructure issues like access to the project repo on GitHub or overriding a firewall for necessary project work, then the abstraction has failed.
Tech Lead Manager (TLM)Tech lead managers (TLMs) are rare in many organizations. In Google, small or nascent teams usually have a TLM who can oversee a group of engineers, guiding them in project execution and ensuring the team’s productivity. This role involves a mix of technical leadership, project management, and people management. You will need a solid technical background to take up this role and should be able to contribute to technical discussions easily. You should be involved in technical design and communicate relevant design decisions to other teams and stakeholders.
TLMs are responsible for setting priorities, resolving technical challenges, and fostering a collaborative team culture. This role offers the opportunity to do both technical execution and people leadership. But it also comes with the challenge of balancing the two areas while not shortchanging either one. To help with this, TLMs will usually have a smaller number of direct reports as compared to engineering managers. TLM responsibilities include the following:
Blending people management with hands-on technical leadership
TLMs must balance their responsibilities as people manager and technical leader. This involves not only overseeing the professional development of the team but also actively participating in the technical aspects of projects, setting an example for team members.
Coach and develop engineers on coding skills
From a people management perspective, part of the TLM’s responsibility is nurturing their team, coaching, providing constructive feedback, and guiding engineers to enhance their technical proficiency. TLMs must also ensure individual contributors are challenged in their work and are on track to reach their personal career goals.
Establish technical standards and architecture
TLMs are responsible for setting technical standards and architecture. This entails defining and maintaining coding practices, architectural principles, design, and code reviews.
Help unblock developers when they are stuck
TLMs play a crucial role in unblocking developers when they encounter challenges. This involves providing technical guidance, removing impediments, and keeping upper management appraised of the project’s progress and resource needs.
Focus on higher-priority technical work
Sometimes, TLMs may need to concentrate on higher-priority technical initiatives. This could even involve hands-on coding or debugging. TLMs may have to delegate specific people management tasks to balance the other demands of their role. This strategic delegation ensures that both aspects of their role receive adequate attention.
Advocate for the team while coordinating cross-functionally
As the advocate for their team, TLMs engage in cross-functional coordination. This includes representing their team’s interests, ensuring effective communication across departments, and fostering collaboration to achieve collective goals.
Make technical decisions weighing various constraints
TLMs are decision makers in technical matters, which involves considering multiple constraints. This includes assessing factors such as project timelines, resource availability, and technical debt to make informed decisions that align with both short-term goals and long-term sustainability.
Provide mentorship and guidance
TLMs play a crucial role in mentoring and guiding team members to enhance their technical skills and professional development. By dedicating time to mentorship, TLMs foster a culture of continuous learning and growth within the team.
As you can tell from the preceding list, having really strong technical aptitude is critical in a TLM role. A TLM often asks intelligent questions and pushes the team to find answers. TLMs communicate a lot with various people, some of whom are purely technical and others of whom are business oriented. TLMs will thus have to switch their communication style constantly. A sign of success as a TLM is effectively balancing all the responsibilities while finding some extra time to write some code occasionally.
While there may be other roles or other names used to refer to these roles among software organizations, I have tried to discuss the key responsibilities of a team leader or manager in an engineering team in this section. However, responsibilities don’t dictate your ability to perform them. How do you know you have what it takes to lead your teams effectively? Find out by assessing yourself on key leadership traits in the next section.
Parting thoughtsIn the preface of the book, Addy outlined who he wrote this book for:
“This book is for engineers wanting to move into leadership roles or engineering leaders who want evidence-based guidance to improve their effectiveness and that of their teams. It is a comprehensive guide to the strategies, frameworks, and best practices that I have found to be most effective in unlocking the full potential of engineering teams and driving transformative results. By sharing real-world examples, practical insights, and actionable advice, I aim to empower you with the tools and knowledge you need to become an exceptional engineering leader in your own right.
At the heart of this book lies a deep exploration of the key traits and behaviors that distinguish highly effective engineers and engineering leaders from their peers. These are the individuals who consistently deliver outstanding results, inspire their teams to reach new heights, and make a lasting impact on the projects and initiatives they lead. By understanding and embodying these characteristics, you, too, can set yourself apart and make a meaningful difference in your role.”
It’s a great time to transition into engineering leadership roles: as there are more and more in-depth resources where engineering leaders like Addy share their hard-earned experience, and way of thinking. Additionally, this book offers a peek at how effective managers at Google operate, and philosophies that are likely to be more common at Google – like the importance of physiological safety, balancing complex interpersonal dynamics, and empowering team members to take ownership of their work.
I hope you enjoyed this deepdive into a more theoretical overview of what we know about effective engineering teams, and a look at how companies like Google think about the TL, EM and TLM roles.
To read on, you can get the book (or e-book.)
And for more reading do check out some of Addy’s other books – including the free e-book titles Software Engineering: The Soft Parts, and The Developer Experience Book. You can also follow Addy on LinkedIn, where he shares learnings on software engineering several times per week.
As related reading, see these past The Pragmatic Engineer articles:
Engineering leadership skillset overlaps: how staff engineers, EMs, PMs, TLMs and TPMs overlap in Big Tech and high-growth startups.
Engineering career paths at Big Tech and scaleups. Levels at Big Tech, the most common software engineering career paths, and what comes after making it to Staff Engineer.
ちょっと急ですが、NIST SP800-63-4 デジタルIDガイドライン第2次公開草案を読む会の第1回を8月30日午後8時より実施します。みなさま奮ってご参加ください。
(8/29現在、随時アップデート中です)
NIST SP800-63-4の第2次公開草案(2pd)が1週間前に公開されました。本日(日本時間29日午前1時〜3時)に行われたワークショップは、計画された一連のワークショップの最初のものであり、最初の公開草案以降の最も重要な変更点のいくつかを概説しています。
Introduction and Housekeeping ワークショップはNIST特別出版物800-63 改訂4第2次公開草案に関するワークショップは、セッションの録画、スライドの入手可能性、質問のためのQ&A機能の使用など、事務連絡から始まりました。[00:00] 本日の議題は以下の通りです: NIST特別出版物800-63-4の概要 このワークショップでは、デジタルアイデンティティガイドラインの第2次公開草案に焦点を当て、主要な変更点、パブリックコメント期間、コメントの提出方法について取り上げました。[02:00] このガイドラインは、連邦政府全体のデジタルアイデンティティ管理に関する基本的な要件を定めており、基本巻とA巻、B巻、C巻の4巻で構成されています。[05:00] 変更の主要な動機 主な動機には、政府サービスへの公平なアクセスの改善、新たな脅威や技術への対応、過去の実装から得られた実世界の教訓の取り込みが含まれます。[07:00] 第1次公開草案における主要な変更点 変更点には、リスク管理の刷新、生体認証要件の更新、新しい本人確認プロセス、そしてプライバシー、使いやすさ、公平性に関する考慮事項が含まれていました。[09:00] タイムラインとパブリックコメント期間 改訂プロセスのタイムラインが確認され、2022年12月の第1次公開草案の発行と2023年8月の第2次公開草案の発行が強調されました。第2次草案のパブリックコメント期間は45日間です。[12:00] Base Volumeの主要な変更 コニー・ラサール氏は、ユーザー管理型ウォレットモデルの導入(第2章)、アイデンティティリスク管理プロセスにおけるサービス「定義」ステップの追加(第3章)、継続的な評価と改善のためのメトリクスの導入、例外事項が起きたときのリドレスメカニズムについて説明しました。[16:00] 特筆すべき点として、IdP(アイデンティティプロバイダー)の変形である利用者管理型ウォレットが導入され、「発行者」がCSP(資格情報サービスプロバイダー)として捉えられるようになりました。 更新されたデジタルアイデンティティのリスク管理プロセスには、オンラインサービスの定義、初期影響評価の実施、そして継続的なリスク評価に基づいたコントロールの調整が含まれています。[20:00] 継続的な評価と改善が強調されており、推奨されるパフォーマンス指標と、問題を公平に扱うための是正措置が提示されています。[25:00] Volume A (Identity Proofing and Enrollment)の主要変更点 デイビッド・タマサク氏は、本人確認の役割と種類の更新、IAL 1(Identity Assurance Level 1)のリバランス、新しいアイデンティティ検証の経路、不正管理要件、および証拠の検証要件の更新について強調しました。[30:00] 本人確認の役割には、本人確認エージェント、信頼できる推薦者、プロセスアシスタント、申請者の参考人が含まれるようになりました。[32:00] IAL 1のバランス調整は、申請者と資格情報サービスプロバイダーにとっての摩擦の低減と選択肢の増加に焦点を当てています。[35:00] AAL 2における新しいアイデンティティ検証の経路には、生体認証を用いないオプションやデジタル証拠の検証が含まれています。[38:00] 新しい不正管理セクションには、資格情報サービスプロバイダーと依拠当事者に対する要件、必須の不正チェック、および不正の疑いがあるケースに関するコミュニケーションチャネルが含まれています。たとえば、死亡日の確認は必須になっています。[42:00] 更新された証拠検証要件には、文書認証システムのパフォーマンス指標や本人確認エージェントのトレーニングが含まれています。[45:00] Volume B: Major Changes (Authenticators and Authentication) アンディ・リーガンチッド氏は、段階的な改良、同期可能な認証器に関する新たな要件、および利用者管理のデジタルアカウントに関するガイドラインの明確化について説明しました。[50:00] 改訂されたアカウント回復セクションでは、アカウント回復プロセスの実装に関してより明確な経路とより高い柔軟性が提供されています。[55:00] パスキーのような同期可能な認証器が現在対応されており、同期ファブリック(sync fabric)に関する追加要件が設けられています。[52:00] デジタルウォレットを認証器として使用することが明確化され、保存された回復コードや信頼できる回復用連絡先など、新しいアカウント回復方法が導入されました。[57:00] Volume C (Federation and Assertions)の主要な変更点 ライアン・ガルーゾ氏は、863 Cの更新された構造、Federation Assurance Level 3(連携保証レベル3)の修正、およびプロトコルベースの例の導入について説明しました。[01:00:00] 新しい構造には、核となる共通の連携要件と、汎用IDPフェデレーションおよびユーザー管理型ウォレットフェデレーションのための別個のセクションが含まれています。[01:02:00] ウォレットモデルにおいて、ウォレットはIdPとしてモデルすることによって取り扱うことができるのでそのようにした(←コメントが受け入れられて嬉しい。) 伝統的なIDPとウォレットとの差として、マルチユーザーかシングルユーザーかで分けている。(←ここはちょっとどうかなと思う。時間の考えも入れたほうが良いのではないか。) 3つめの変更点は、Bound Authenticators の導入。連携保証レベル3には、現在、鍵保持者(Holder of Key)のアサーションとバインドされた認証器が含まれています。[01:05:00] プロトコルベースの例も追加した。OpenID ConnectやSAMLなどの連携プロトコルを実装するための高レベルな説明を提供しています。(←バックチャンネルを使わなくてもFAL2実現できるというコメントがQ&Aにあった。実際、response type = id_token ではできるはず。iGove WG でFAL2プロファイルとか作るとよいのかもしれない。)[01:08:00] パブリックコメント期間と次のステップ パブリックコメント期間は10月7日に締め切られます。コメントはメールまたはExcelスプレッドシートを使用して提出できます。最終決定までの時間は、受け取ったコメントの量によって変わります。[01:15:00] チームは一般からのフィードバックの重要性を強調し、レビュープロセスへの参加を奨励しました。[01:20:00] 特に以下の分野についてフィードバックを求めています: これが最後のパブリックコンサルテーションとなり、新年に出版が予定されています。In response to a post on X about China's social credit system, Paul Conlon said:
Digital ID is ultimately about access control where those who impose the system are the ones determining what you are required to be and do.
Provision of resources and liberties become conditional upon the whims of the affluent. Doesn't sound safe or convenient to me.
From X
Referenced 2024-08-28T08:10:31-0400
How Paul said this struck me because I've been thinking a lot about access control lately. I believe that we build identity systems to manage relationships, but, as Paul points out, the ultimately utility of identity systems in many cases is access control.
This isn't, by itself, a bad thing. I'm glad that Google controls access to my GMail account so that only I can use it. But it doesn't stop there. If I use my Google account to log into other things, then Google ultimately controls my access to everything I've used it for. This is federation's original sin1.
Paul's comment points out the primary problem with how we build identity systems today: when access control is centralized, it inherently shifts power towards those who manage the system. This dynamic can lead to a situation where individuals must conform to the expectations or demands of those in control, just to maintain their access to essential services or resources. While we often accept this trade-off for convenience—like using Google to manage multiple logins—the broader implications are troubling.
The more we rely on federated identity systems, with their tendency to centralization, the more we risk ceding control over our digital lives, reducing our autonomy and increasing our dependence on entities whose goals may not align with our own. This is why the principles of self-sovereign identity (SSI) are so compelling. SSI proposes a model where individuals maintain control over their own identity, reducing the risks associated with centralized access control and enhancing personal freedom in the digital realm.
Critics of SSI will claim that giving people control over their identity means we have to accept their self assertions. Nothing could be further from the truth. When someone wants me to prove I'm over 18, I use a driver's license. The state is asserting my age, not me. But I'm in control of who I show that to and where. Sovereignty is about borders and imposes a system of relationships.
Now, China could use SSI to build the social credit system. One credential, controlled by the state, that is used to access everything. SSI makes individual control structurally possible, but can’t guarantee it. Technology alone can't solve this problem. As a society, we have to want a digital world, modeled on the physical one, where individuals are the locus of control and use information and assertions from a variety of credentials to build and and interact in peer-to-peer relationships. Until we value freedom and independence in the digital world, we will yield up control o