Identity Blog Catcher

json-flatten, now with format documentation

json-flatten is a fun little Python library I put together a few years ago for converting JSON data into a flat key-value format, suitable for inclusion in an HTML form or query string. It lets you take a structure like this one:

{"foo": {"bar": [1, True, None]}

And convert it into key-value pairs like this:

foo.bar.[0]$int=1 foo.bar.[1]$bool=True foo.bar.[2]$none=None

The flatten(dictionary) function function converts to that format, and unflatten(dictionary) converts back again.

I was considering the library for a project today and realized that the 0.3 README was a little thin - it showed how to use the library but didn't provide full details of the format it used.

On a hunch, I decided to see if files-to-prompt plus LLM plus Claude 3.5 Sonnet could write that documentation for me. I ran this command:

files-to-prompt *.py | llm -m claude-3.5-sonnet --system 'write detailed documentation in markdown describing the format used to represent JSON and nested JSON as key/value pairs, include a table as well'

That *.py picked up both json_flatten.py and test_json_flatten.py - I figured the test file had enough examples in that it should act as a good source of information for the documentation.

This worked really well! You can see the first draft it produced here.

It included before and after examples in the documentation. I didn't fully trust these to be accurate, so I gave it this follow-up prompt:

llm -c "Rewrite that document to use the Python cog library to generate the examples"

I'm a big fan of Cog for maintaining examples in READMEs that are generated by code. Cog has been around for a couple of decades now so it was a safe bet that Claude would know about it.

This almost worked - it produced valid Cog syntax like the following:

[[[cog example = { "fruits": ["apple", "banana", "cherry"] } cog.out("```json\n") cog.out(str(example)) cog.out("\n```\n") cog.out("Flattened:\n```\n") for key, value in flatten(example).items(): cog.out(f"{key}: {value}\n") cog.out("```\n") ]]] [[[end]]]

But that wasn't entirely right, because it forgot to include the Markdown comments that would hide the Cog syntax, which should have looked like this:

<!-- [[[cog --> ... <!-- ]]] --> ... <!-- [[[end]]] -->

I could have prompted it to correct itself, but at this point I decided to take over and edit the rest of the documentation by hand.

The end result was documentation that I'm really happy with, and that I probably wouldn't have bothered to write if Claude hadn't got me started.

Tags: claude-3-5-sonnet, llm, anthropic, claude, ai, llms, ai-assisted-programming, generative-ai, projects, json

こんにちは、富士榮です。
ウォレットの話が続きます。

Ott Sarv氏がLInkedInに投稿した記事ですがCC BY4.0のライセンスで公開されていますのでこちらで読んでいこうと思います。 https://www.linkedin.com/pulse/opinion-critical-reflection-future-openwallet-light-eidas-ott-sarv-jredf/

なお、 黄色マーカーは私によります 赤字は私のコメントです
さて、早速みていきます。 意見:eIDAS 2.0を踏まえたオープンウォレットの将来に関する重要な考察 デジタルIDソリューションのエンド・ツー・エンド・アーキテクトとして、ヨーロッパ、東南アジア、インドと中国の間の地域、アフリカ大陸などの多様な地域で活動する中で、私はデジタルIDシステムの導入に伴う大きな影響と重大な課題を目の当たりにしてきました。その経験から、信頼はデジタルIDの基本要素であり、その信頼はしばしば政府の権限や規制監督と密接に結びついていることを学びました。最近、eIDAS 2.0 ドラフト仕様が発表され、規制の状況はデジタル ID フレームワークに対する 政府の管理を強化する方向にシフトしていることがますます明らかになっている。このシフトは、ステートレスでオープンソースのデジタル・ウォレットを作成することを目的とする OpenWallet Foundation のようなイニシアチブの将来について重大な問題を提起している。
→興味深いですね。政府などの機関に信頼の基点を置く必要が叫ばれつつもウォレットにアイデンティティ情報を格納して持ち運ぶ、というところにこれまで「自己主権」ということを叫んできた人たちにとっては矛盾を産み始めているのかもしれません。
規制の背景を理解する： デジタルアイデンティティにおける政府の役割 ヨーロッパの規制環境から東南アジアやアフリカのダイナミックなデジタル・ ランドスケープまで、私が働いてきたどの地域でも、デジタル ID ソリューションの信頼確立に おける政府の中心的役割は一貫した要因であった。インドや中国の国家 ID プログラムであろうと、アフリカの国家が支援するデジタル ID 枠組みであろうと、政府の関与は大量採用と信頼の達成に不可欠であった。 これらの経験は、デジタル ID システムが成功するためには、その妥当性とセキュリ ティを保証する権威ある情報源（通常は政府）が必要であるという重大な現実を浮き彫りにし ている。政府は個人データおよび公共の利益の主要な管理者と見なされるため、個人は政府によって支 持されるデジタル ID を信頼する。政府の承認または認識がなければ、多くのデジタル ID イニシアチブ は、その技術革新にかかわらず、牽引力を得るのに苦労する。
→適用領域次第だろうとは思いますが、政府を含む権威のある情報ソースがデジタルアイデンティティに関する信頼の基点になりやすいのは事実だと思います。一方でこのことが先日のID Dayの話のように国家にネグレクトされた人たちが存在する要因の一つにもなっていることも事実です。やはり人類は「信頼」について深く考察すべき時期に来ているんじゃないかとおもます。 Open Walletの課題： eIDAS 2.0規制への対応 分散型デジタル・ウォレットというビジョンを掲げるOpen Wallet Foundationは、この文脈において大きな課題に直面している。eIDAS 2.0 仕様草案は、欧州連合内のデジタル ID は国家発行または国家承認でなければならず、こ れらの ID に対する信頼は本質的に政府の権威と結びついていると明言している。この枠組みは、デジタル ID システムに対する国の管理を強化する広範な傾向を反映し ている。 Open Walletにとって、この規制環境は重大なジレンマをもたらす。オープンソースの原則と包括的な開発へのコミットメントは賞賛に値するが、その現在のアプローチは、政府主導の枠組みへの準拠が不可欠なEUのような主要市場の規制の現実と一致しない可能性がある。国家が支援するデジタル・アイデンティティ・システムへの移行は、特に政府規制の遵守が義務付けられている法域では、ステートレス・モデルの余地が限られている可能性があることを示唆している。
→まさに先に書いた通りです。分散型でオープン性を掲げる一方で政府に信頼の基点を置かざるを得ない、というのは矛盾する可能性があります。
Open Wallet Foundationへの戦略的提言 このような課題を踏まえて、Open Wallet Foundationに対する戦略的提言をいくつか紹介します： A. プロジェクトの戦略的方向性の再評価 進化する規制の状況を考えると、Open Wallet Foundationは現在のモデルの限界を認識する時かもしれない。eIDAS 2.0や同様のフレームワークで政府が支援するデジタルIDが世界的に重視されていることから、Open Walletは現在のプロジェクトを終了し、戦略を見直すことを検討すべきです。これは失敗を意味するのではなく、当初のビジョンが現在の状況では実現不可能かもしれないという現実的な認識を意味する。
→なかなか刺激的な提言ですね。行き過ぎな気はします。あくまで実装としてのOpen Walletと政府に信頼の基点を置くクレデンシャルは両立できるんじゃないのか？とも思います。
B. 政府間協力への軸足 Open Walletは、ステートレス・デジタル・ウォレット・モデルを継続する代わりに、政府との協力によって既存のデジタル ID フレームワークを強化する方向に軸足を移すことができる。これには、政府主導のデジタル ID イニシアチブを補完しサポートするオープンソースのツール、モジュール、または標準を開発することが含まれます。政府の要求と連携することで、オープンウォレットはイノベーションを促進しながら、グローバルなデジタルIDエコシステムにおいて関連性を保つことができます。
→これは一部でやってるんじゃないの？とも思いますが、ちゃんと政府の方向性とコンフリクトがない形を目指していきましょう、ってところですね。
C. イノベーションとコンプライアンスのバランスをとるハイブリッドモデルの提唱 OpenWallet は、その包括的なマルチステークホルダーアプローチと規制遵守の必要性のバランスをとるハイブリッドモデルを模索すべきです。eIDAS 2.0のような規制の遵守に焦点を当てた専門ワーキンググループを結成することは、オープン性へのコミットメントを維持しながら、OpenWalletのビジョンを法的要件と整合させるのに役立つでしょう。EUの規制当局や他の標準化団体と直接関わることで、貴重な洞察や指針が得られるだろう。
→まぁ、そうなりますよね。戦っても仕方ない話なので、前項にもある通り歩調を合わせるっていうことが必要になりそうです。今更ながらですがマルチステークホルダーといっている中の重要な一部として政府も入っている、ということです。
D. 主要ステークホルダーとの継続的な対話の促進 関連性を維持するために、Open Walletは規制当局、政策立案者、およびその他の主要な利害関係 者と継続的に対話する必要があります。協議に参加し、フィードバックを提供し、ベストプラクティスを共有することで、Open Walletはデジタル ID 標準の進化に貢献する貴重な存在として位置付けられ、新たな規制の動向に 応じて戦略を適応させることができます。
→前項までと一緒ですね。ある程度歩調は合わせていると思いますが、もっと明確にやっていくべきなのかもしれません。 デジタルアイデンティティ戦略の再編 エンド・ツー・エンドのデジタル・アイデンティティの専門家として、私はOpen Wallet Foundationとその他の関係者に対し、進化する規制の状況に照らして現在のアプローチを批判的に評価するよう強く求めます。デジタルIDの信頼の礎としての政府の役割を認識し、オープンソースのイノベーションを国家主導のフレームワークと連携させる新たな方法を模索することが不可欠です。 私は、デジタル ID コミュニティに対し、政府および規制当局との協力関係を強化し、革新的で、準拠 性が高く、安全で、広く受け入れられるデジタル ID ソリューションを開発するための共通基盤を見出すよう呼びかける。
→良い投げ込みですね。こういうことを関係者がちゃんと意識をする良いきっかけになると良いと思います。これはもちろん日本においても、です。
デジタル・アイデンティティの現実的な道筋 デジタル ID の将来は、革新、信頼、および規制の間の微妙なバランスによって形成される。eIDAS 2.0 ドラフト仕様が示すように、デジタル ID の信頼は依然として政府の権限と監視と密接に関係 している。Open Walletのようなイニシアチブにとって、この現実は戦略的な再評価を必要とする。 現在のプロジェクトを終了し、規制の枠組みに沿ったモデルに軸足を移すことで、Open Walletはデジタル ID エコシステムに有意義な貢献を続けることができる。このアプローチにより、デジタル ID ソリューションは、規制の期待を尊重しながらも、多様な地域の ユーザーのニーズを満たし、インパクトがあり、コンプライアンスがあり、世界的に適切なものとなる。
→Open Wallet Foundationがどうしていくべきか、については必要以上にここでは触れませんが、技術だけではダメで、規制、そしてそもそも「信頼」はどのようにして醸成されるのか？を考えていけると良いと思います。（これはウォレットに限らず全てのデジタルIDシステムについて言えることですが）

ということで興味深く読ませていただきました。

Docker images using uv's python

Michael Kennedy interviewed uv/Ruff lead Charlie Marsh on his Talk Python podcast, and was inspired to try uv with Talk Python's own infrastructure, a single 8 CPU server running 17 Docker containers (status page here).

The key line they're now using is this:

RUN uv venv --python 3.12.5 /venv

Which downloads the uv selected standalone Python binary for Python 3.12.5 and creates a virtual environment for it at /venv all in one go.

Via @mkennedy

Tags: docker, uv, python, charlie-marsh

More serendipity

No surprise that a Fowles quote surfaced this morning, he is after all one of my favorites.

The surprise lay in the relevance to a conversation with friends last night.

Topic

How society and norms morph over time, by dint of individual actions.

First learning, then adapting followed by experimentation and pushing boundaries to ultimately creating.

🖇️ Talking about knowing what you’re talking about - nobody who knows what they are talking about would confuse me with a real programmer.

That said, I just read 🔗 Going the Extra Mile — Beyond CSS, most of which is well above my technical pay grade - but still I was fascinated by the attention to detail around ‘crafting Craft’.

Consider - a whole post (well 75%) devoted to the process behind getting a check box ‘just right’ and the remaining 25% explaining ‘push away’, which I knew had to be coded in some way - but who knew there is a phrase for it?

Great read - even if you aren’t technical you get the idea of their focus on good design - Steve would have been proud of them

“When you’re a carpenter making a beautiful chest of drawers, you’re not going to use a piece of plywood on the back, even though it faces the wall and nobody will ever see it. You’ll know it’s there, so you’re going to use a beautiful piece of wood on the back. For you to sleep well at night, the aesthetic, the quality, has to be carried all the way through.”

💬 Steve Jobs

… which probably explains why they won app of the year and were name called (not once - but twice) in this year’s WWDC.

The article ends with …

All of this for what? Will all users see the difference? — Probably not. Was it worth it? — Absolutely. These are the minor details that often go unnoticed when present, but become conspicuous when missing. We strive for having as many of those details as possible, making the experience great but at the same time — seamless.

Kudos to Craft - a Great App and a serious tool for writing documents in the 21st Century.

( 🖇️ Related (Page-less Writing Apps) )

Read and agree with a lot of the @gruber post 🔗 The iOS Continental Drift Widens

After I read it - I jumped to my Readwise queue - and up pops this.

My reading habits are clearly being tracked.

🔗 Lex raises $2.75M for its AI writing tool that helps writers get past blocks.

The AI stuff is cool, but there was one thing about this app that made me want to keep using it: It has no historical baggage. I find it odd that many modern word processors like Google Docs and Word retain a bias toward pagination — the UX is intended for printing documents onto letter-sized paper. Lex, in contrast, does away with all that.

💬 TechCrunch

Been exploring 🔗 Lex.page for a couple ‘ months. I even pay - but not yet used it enough to know whether I am going to continue to pay.

That said, the above quote is interesting and I agree with the ‘historical baggage thought - but they are hardly at the vanguard of that kind of thinking!

It as if the writer has never heard of;

Craft Ulysses IAWriter Drafts, et al

Let alone, Obsidian, Roam and Notion

Side note - Craft. Most definitely a personal favorite. And I am increasingly using it to share thoughts, ideas, minutes, reports to specific audiences - most often under ‘loose’ password control.

Cant wait to get my hands on the new stuff they have in Beta - though even that still wont solve what I really really want!

Adding to which their own foray into AI support, I just want to tie apps like that into a single world - and not have different versions of different LLMs spread around various apps. ANNOYING.

Via John Naughton

I recently posted on my experience with using a few Python examples of retrieval augmented generation (RAG) with an LLM (OpenAI API). They were underwhelming, though they did provide a lot of insight for me. This post is about my next experiment: let’s get AI tools to generate examples for me. The results were mixed but not for reasons you might think. Here’s what I found using ChatGPT, Microsoft Copilot and Cursor.

Note: you can find the generated code in my LLM_Generated_RAG Github repo. I will likely add to this repo as I try new prompts, LLMs, and approaches.

ChatGPT 4o

Let’s start with the most well-known, ChatGPT. I used a simple prompt to see what it would do. I didn’t provide prompts for different parts of the code. It should not be very extensive code so I didn’t feel the need to divide it into separate prompts. If you want to see my very elementary prompt pull up the ChatGPT related file in the repo.

The first thing I noticed is that it was using PyPDF2 which was deprecated in Decemeber of 2022. Not surprising as, almost certainly, a lot of what the model was trained on used PyPDF2. It actually created well-structured code with functions for the different discrete operations. What I realized later, due to the output of the Cursor tool, is that it created a more granular set of code than it had to. This is exactly what I was looking for but didn’t specify in the prompt. What do I mean? It did the following:

read the PDF and extracted the text split the document into chunks created embeddings for the chunks using the OpenAI text-embedding-ada-002 created an embedding for the query searched the chunks using faiss generated a response using the query and the chunk(s) as context

This is what I wanted. Turns out there was a very different way to interpret the prompt.

The downside. The code didn’t run out the box. Also, I believe it only used one chunk for the context to send along with the query. I have to investigate the code more closely to be sure on that single chunk thing. The embedding search chunk function I need to investigate more to understand it. I didn’t try to make the code run for now as that was not part of the experiment. I expect I will try to modify it soon to function. I’ll report back the results.

This granular set of actions was very different than what I got from Cursor.

Cursor

The Cursor prompt was not exactly the same as what I used for ChatGPT, which was bad of me to do. I got a little lazier, but the crux was the same. You can see that prompt here.

The results were very different. It did not provide the granular steps that ChatGPT did. It met the requirement more succinctly.

extract the text from the PDF, also using PyPDF2 pass the entire text to ChatGPT as context along with the prompt

It ran out of the gate. Hypothetically, it would return the expected results as well. It would work similarly to how one does it in the ChatGPT interface. Not what I wanted, but I hadn’t specified to break it down so that I was creating embeddings etc. Fair game.

The downside, the context was too large (43,000+ tokens) for the tokens per minute limit of my OpenAPI account level. So I didn’t get to see the output, but have no doubt it would have produced similar results as if I had done it through the ChatGPT user interface.

Microsoft Copilot

What can I say? I don’t know what I would have gotten here because Copilot blocked the output. This is apparently because “GitHub Copilot checks code completion suggestions with their surrounding code of about 150 characters against public code on GitHub. If there is a match, or a near match, the suggestion is not shown to you.”

There is a flag you can set in your Github settings to turn this on or off. I checked and my flag and it is set to “Allowed” but you see the results.

I’ll continue to try to troubleshoot this issue and see if I can get something out of Copilot and update this post if I do.

In recap

In recap. One provided the detail I wanted even though I hadn’t specified it, a happy coincidence, but it did not run out of the gate. The second took a very different approach and would have provided the desired results in LLM response if my OpenAPI account level had supported a large enough tokens per minute for 43,000+ tokens in the context. But it wasn’t the code I was hoping for. Copliot just didn’t work for me for reasons I don’t yet understand. More experimentation to come.

If a set of accounts live in isolation forever, what happens to those accounts only matters within that isolated system. But when we make a bubble in our federated network, we aren’t just making a stand-alone system that can go live in a fully disconnected state. Over time, we expect things to re-connect, and when that happens, data needs to be harmonized across the boundaries.

So many bubbles, so many connections Data Synchronization

When multiple independent systems live together in the world and share data, inevitably that data is going to get out of sync. In a purely heirarchical system, we’re mostly concerned with building a consensus around the correct state of the shared data. We see this approach in distributed ledger systems, where nodes eventually come to a common view of what the shared state of the world ought to be.

But we don’t have that in a bubble-based architecture, because we don’t expect everyone to have the same view. Instead, we expect many different independent views to contribute to each other in a distributed fashion. This effectively means that for each bubble, it can send updates in and out to other systems. In most cases, there is a directionality to the data flow: one side is going to be treated as more authoritative than the other for a given context. What a bubble does when it’s on either end of that gap changes how we view the synchronization.

Account Updates From Above

When changes are pushed to us from an authoritative source, the simplest thing is to overwrite everything. After all, if it’s authoritative, why wouldn’t we just take that statement as truth? But the reality is that we’ve likely augmented our record of that user with additional details, overrides, and other localized changes that we don’t want to lose.

Local data (blue) shadows updates from the source (orange), in some cases

In these cases, we can shadow the data. In other words, we keep a copy of the source’s data separate from our own local view. When we get an update from that source, we can update our copy of the source’s data with wild abandon. We can then decide, by local policy, whether we want to adjust our overrides based on the update. Importantly, this decision is separate from accepting and processing the updated data from the source. This setup allows us to keep local information in the bubble at the same time that we sync from elsewhere.

Account Updates From Below

An authoritative bubble is sometimes going to want to pull updated information from the bubbles that live downstream. These are systems that we’ve sent accounts out to, and those systems might have something to tell us about our users. Maybe there’s an additional accreditation that’s been earned, or something to represent additional access details outside of our bubble, or even just an update to one of the core fields we sent down.

In any case, the downstream bubble is sending us additional data about a user, and we now have the chance to do something about it. If nothing else, we can store it and note it. If we want to, we can update the user’s record that we hold locally, and even go so far as to propagate that downward again to other bubbles.

Changing Trust

It’s not just user data that we can pass around, though that’s the most common piece we’d expect to see. The bubbles can also pass about other bubbles to each other, and incorporate what they learn into their own systems.

For many situations that fit the bubble architecture patterns, we expect to meet new peers in the field and make new decisions based on local context and requirements. These newly-discovered bubbles can then be propagated through the network, along with potentially interesting information about what the presenting bubble trusts it for.

Audit Logs

Now that we can identify users and the systems they come from, we can start to do one of the most interesting and perhaps complex jobs of a reconnected bubble: audit logs. When a bubble gets provisioned, that provisioning authority is likely to want to know what happens in that bubble during the disconnected spells. The bubble can package up the relevant audit log history and pass it along to the authorities that need it.

But auditing can go further than that: for any user that enters our bubble from somewhere else, we probably want to report back to the system that sent them to us. Since we know where we originally learned of them, and we know how to identify that system as a whole, we can filter and target the information we’re sending over. And we can do this while processing the updates they’re sending us about the account.

Conclusion

The bubble pattern sits between the always-connected and always-separate models, but the moment of reconnection gives us a profound opportunity to process user and trust data in ways that we may have never considered.

Datasette 1.0a16

This latest release focuses mainly on performance, as discussed here in Optimizing Datasette a couple of weeks ago.

It also includes some minor CSS changes that could affect plugins, and hence need to be included before the final 1.0 release. Those are outlined in detail in issues #2415 and #2420.

Tags: projects, datasette

New improved commit messages for scrape-hacker-news-by-domain

My simonw/scrape-hacker-news-by-domain repo has a very specific purpose. Once an hour it scrapes the Hacker News /from?site=simonwillison.net page (and the equivalent for datasette.io) using my shot-scraper tool and stashes the parsed links, scores and comment counts in JSON files in that repo.

It does this mainly so I can subscribe to GitHub's Atom feed of the commit log - visit simonw/scrape-hacker-news-by-domain/commits/main and add .atom to the URL to get that.

NetNewsWire will inform me within about an hour if any of my content has made it to Hacker News, and the repo will track the score and comment count for me over time. I wrote more about how this works in Scraping web pages from the command line with shot-scraper back in March 2022.

Prior to the latest improvement, the commit messages themselves were pretty uninformative. The message had the date, and to actually see which Hacker News post it was referring to, I had to click through to the commit and look at the diff.

I built my csv-diff tool a while back to help address this problem: it can produce a slightly more human-readable version of a diff between two CSV or JSON files, ideally suited for including in a commit message attached to a git scraping repo like this one.

I got that working, but there was still room for improvement. I recently learned that any Hacker News thread has an undocumented URL at /latest?id=x which displays the most recently added comments at the top.

I wanted that in my commit messages, so I could quickly click a link to see the most recent comments on a thread.

So... I added one more feature to csv-diff: a new --extra option lets you specify a Python format string to be used to add extra fields to the displayed difference.

My GitHub Actions workflow now runs this command:

csv-diff simonwillison-net.json simonwillison-net-new.json \ --key id --format json \ --extra latest 'https://news.ycombinator.com/latest?id={id}' \ >> /tmp/commit.txt

This generates the diff between the two versions, using the id property in the JSON to tie records together. It adds a latest field linking to that URL.

The commits now look like this:

Tags: shot-scraper, github-actions, projects, hacker-news, git-scraping, json

I've been having a bunch of fun taking advantage of CORS-enabled LLM APIs to build client-side JavaScript applications that access LLMs directly. I also span up a new Datasette plugin for advanced permission management.

LLMs from client-side JavaScript Converting PDFs to HTML and Markdown Adding some class to Datasette forms On the blog Releases TILs LLMs from client-side JavaScript

Anthropic recently added CORS support to their Claude APIs. It's a little hard to use - you have to add anthropic-dangerous-direct-browser-access: true to your request headers to enable it - but once you know the trick you can start building web applications that talk to Anthropic's LLMs directly, without any additional server-side code.

I later found out that both OpenAI and Google Gemini have this capability too, without needing the special header.

The problem with this approach is security: it's very important not to embed an API key attached to your billing account in client-side HTML and JavaScript for anyone to see!

For my purposes though that doesn't matter. I've been building tools which prompt() a user for their own API key (sadly restricting their usage to the tiny portion of people who both understand API keys and have created API accounts with one of the big providers) - then I stash that key in localStorage and start using it to make requests.

My simonw/tools repository is home to a growing collection of pure HTML+JavaScript tools, hosted at tools.simonwillison.net using GitHub Pages. I love not having to even think about hosting server-side code for these tools.

I've published three tools there that talk to LLMs directly so far:

haiku is a fun demo that requests access to the user's camera and then writes a Haiku about what it sees. It uses Anthropic's Claude 3 Haiku model for this - the whole project is one terrible pun. Haiku source code here. gemini-bbox uses the Gemini 1.5 Pro (or Flash) API to prompt those models to return bounding boxes for objects in an image, then renders those bounding boxes. Gemini Pro is the only of the vision LLMs that I've tried that has reliable support for bounding boxes. I wrote about this in Building a tool showing how Gemini Pro can return bounding boxes for objects in images. Gemini Chat App is a more traditional LLM chat interface that again talks to Gemini models (including the new super-speedy gemini-1.5-flash-8b-exp-0827). I built this partly to try out those new models and partly to experiment with implementing a streaming chat interface agaist the Gemini API directly in a browser. I wrote more about how that works in this post.

Here's that Gemini Bounding Box visualization tool:

All three of these tools made heavy use of AI-assisted development: Claude 3.5 Sonnet wrote almost every line of the last two, and the Haiku one was put together a few months ago using Claude 3 Opus.

My personal style of HTML and JavaScript apps turns out to be highly compatible with LLMs: I like using vanilla HTML and JavaScript and keeping everything in the same file, which makes it easy to paste the entire thing into the model and ask it to make some changes for me. This approach also works really well with Claude Artifacts, though I have to tell it "no React" to make sure I get an artifact I can hack on without needing to configure a React build step.

Converting PDFs to HTML and Markdown

I have a long standing vendetta against PDFs for sharing information. They're painful to read on a mobile phone, they have poor accessibility, and even things like copying and pasting text from them can be a pain.

Complaining without doing something about it isn't really my style. Twice in the past few weeks I've taken matters into my own hands:

Google Research released a PDF paper describing their new pipe syntax for SQL. I ran it through Gemini 1.5 Pro to convert it to HTML (prompts here) and got this - a pretty great initial result for the first prompt I tried! Nous Research released a preliminary report PDF about their DisTro technology for distributed training of LLMs over low-bandwidth connections. I ran a prompt to use Gemini 1.5 Pro to convert that to this Markdown version, which even handled tables.

Within six hours of posting it my Pipe Syntax in SQL conversion was ranked third on Google for the title of the paper, at which point I set it to <meta name="robots" content="noindex> to try and keep the unverified clone out of search. Yet more evidence that HTML is better than PDF!

I've spent less than a total of ten minutes on using Gemini to convert PDFs in this way and the results have been very impressive. If I were to spend more time on this I'd target figures: I have a hunch that getting Gemini to return bounding boxes for figures on the PDF pages could be the key here, since then each figure could be automatically extracted as an image.

I bet you could build that whole thing as a client-side app against the Gemini Pro API, too...

Adding some class to Datasette forms

I've been working on a new Datasette plugin for permissions management, datasette-acl, which I'll write about separately soon.

I wanted to integrate Choices.js with it, to provide a nicer interface for adding permissions to a user or group.

My first attempt at integrating Choices ended up looking like this:

The weird visual glitches are caused by Datasette's core CSS, which included the following rule:

form input[type=submit], form button[type=button] { font-weight: 400; cursor: pointer; text-align: center; vertical-align: middle; border-width: 1px; border-style: solid; padding: .5em 0.8em; font-size: 0.9rem; line-height: 1; border-radius: .25rem; }

These style rules apply to any submit button or button-button that occurs inside a form!

I'm glad I caught this before Datasette 1.0. I've now started the process of fixing that, by ensuring these rules only apply to elements with class="core" (or that class on a wrapping element). This ensures plugins can style these elements without being caught out by Datasette's defaults.

The problem is... there are a whole bunch of existing plugins that currently rely on that behaviour. I have a tricking issue about that, which identified 28 plugins that need updating. I've worked my way through 8 of those so far, hence the flurry of releases listed at the bottom of this post.

This is also an excuse to revisit a bunch of older plugins, some of which had partially complete features that I've been finishing up.

datasette-write for example now has a neat row action menu item for updating a selected row using a pre-canned UPDATE query. Here's an animated demo of my first prototype of that feature:

On the blog

anthropic

Claude's API now supports CORS requests, enabling client-side applications - 2024-08-23 Explain ACLs by showing me a SQLite table schema for implementing them - 2024-08-23 Musing about OAuth and LLMs on Mastodon - 2024-08-24 Building a tool showing how Gemini Pro can return bounding boxes for objects in images - 2024-08-26 Long context prompting tips - 2024-08-26 Anthropic Release Notes: System Prompts - 2024-08-26 Alex Albert: We've read and heard that you'd appreciate more t... - 2024-08-26 Gemini Chat App - 2024-08-27 System prompt for val.town/townie - 2024-08-28 How Anthropic built Artifacts - 2024-08-28 Anthropic's Prompt Engineering Interactive Tutorial - 2024-08-30 llm-claude-3 0.4.1 - 2024-08-30

ai-assisted-programming

Andy Jassy, Amazon CEO: [...] here’s what we found when we integrated [Am... - 2024-08-24 AI-powered Git Commit Function - 2024-08-26 OpenAI: Improve file search result relevance with chunk ranking - 2024-08-30 Forrest Brazeal: I think that AI has killed, or is about to kill, ... - 2024-08-31

gemini

SQL Has Problems. We Can Fix Them: Pipe Syntax In SQL - 2024-08-24 NousResearch/DisTrO - 2024-08-27

python

uvtrick - 2024-09-01 Anatomy of a Textual User Interface - 2024-09-02 Why I Still Use Python Virtual Environments in Docker - 2024-09-02 Python Developers Survey 2023 Results - 2024-09-03

security

Top companies ground Microsoft Copilot over data governance concerns - 2024-08-23 Frederik Braun: In 2021 we [the Mozilla engineering team] found “... - 2024-08-26 OAuth from First Principles - 2024-09-05

projects

My @covidsewage bot now includes useful alt text - 2024-08-25

armin-ronacher

MiniJinja: Learnings from Building a Template Engine in Rust - 2024-08-27

ethics

John Gruber: Everyone alive today has grown up in a world wher... - 2024-08-27

open-source

Debate over “open source AI” term brings new push to formalize definition - 2024-08-27 Elasticsearch is open source, again - 2024-08-29

performance

Cerebras Inference: AI at Instant Speed - 2024-08-28

sqlite

D. Richard Hipp: My goal is to keep SQLite relevant and viable thr... - 2024-08-28

aws

Leader Election With S3 Conditional Writes - 2024-08-30

javascript

Andreas Giammarchi: whenever you do this: `el.innerHTML += HTML` ... - 2024-08-31

openai

OpenAI says ChatGPT usage has doubled since last year - 2024-08-31

art

Ted Chiang: Art is notoriously hard to define, and so are the... - 2024-08-31

llm

anjor: `history | tail -n 2000 | llm -s "Write aliases f... - 2024-09-03

vision-llms

Qwen2-VL: To See the World More Clearly - 2024-09-04 Releases datasette-import 0.1a5 - 2024-09-04
Tools for importing data into Datasette datasette-search-all 1.1.3 - 2024-09-04
Datasette plugin for searching all searchable tables at once datasette-write 0.4 - 2024-09-04
Datasette plugin providing a UI for executing SQL writes against the database datasette-debug-events 0.1a0 - 2024-09-03
Print Datasette events to standard error datasette-auth-passwords 1.1.1 - 2024-09-03
Datasette plugin for authentication using passwords datasette-enrichments 0.4.3 - 2024-09-03
Tools for running enrichments against data stored in Datasette datasette-configure-fts 1.1.4 - 2024-09-03
Datasette plugin for enabling full-text search against selected table columns datasette-auth-tokens 0.4a10 - 2024-09-03
Datasette plugin for authenticating access using API tokens datasette-edit-schema 0.8a3 - 2024-09-03
Datasette plugin for modifying table schemas datasette-pins 0.1a4 - 2024-09-01
Pin databases, tables, and other items to the Datasette homepage datasette-acl 0.4a2 - 2024-09-01
Advanced permission management for Datasette llm-claude-3 0.4.1 - 2024-08-30
LLM plugin for interacting with the Claude 3 family of models TILs Testing HTML tables with Playwright Python - 2024-09-04 Using namedtuple for pytest parameterized tests - 2024-08-31

Tags: css, javascript, pdf, projects, ai, datasette, weeknotes, generative-ai, llms, anthropic, claude, gemini, claude-3-5-sonnet, cors

OAuth from First Principles

Rare example of an OAuth explainer that breaks down why each of the steps are designed the way they are, by showing an illustrative example of how an attack against OAuth could work in absence of each measure.

Ever wondered why OAuth returns you an authorization code which you then need to exchange for an access token, rather than returning the access token directly? It's for an added layer of protection against eavesdropping attacks:

If Endframe eavesdrops the authorization code in real-time, they can exchange it for an access token very quickly, before Big Head's browser does. [...] Currently, anyone with the authorization code can exchange it for an access token. We need to ensure that only the person who initiated the request can do the exchange.

Via Hacker News

Tags: security, oauth

こんにちは、富士榮です。
みんな大好き”ウォレット”ですが、Verifiable CredentialsやmDocについて語る際に”ウォレット”を持ち出してしまうと、その部分の抽象化レベルだけ他と違ってしまって急に訳がわからない話になってしまうなぁ、、という悩みがあります。


みなさん”ウォレット”と聞くと「スマホにインストールされたネイティブアプリ」を想起してしまうからかもしれません。本来はウォレットの実装方式ではなく、クレデンシャルの保有者（Holder）に着目しないといけないんですけどね。
※つまり、「Issuer-Holder-Verifier」という3パーティモデル（IHVモデル）の話をしているのに、途中から「Issuer-Wallet-Verifier」というレベル感が合わない話にすり替わってしまうという話です。
この悩みにも関連しますが、我らがアンディー（Andrew Hindle）が良いコラムを書いていたので紹介しておきましょう。
Identity Wallets as Infrastructure - Andrew Hindle https://www.hindleconsulting.com/posts/wallets_as_infrastructure/
一言でまとめると 「ウォレットを機能として考えるのではなくインフラコンポーネントとして考えましょう。最終的にウォレット・プロバイダーを選択することなく携帯電話事業者などから提供されるデフォルトのウォレットを使うことになるでしょう」 という話です。
私もたまに「ウォレットの乱立って今後どうなるの？」って聞かれますが、最近は「ウォレットとして独立したアプリケーションとして捉える時代は終わるんじゃない？もっと上位サービスの中に自然と存在している状態になって見えなくなるっていうのがユーザーにとって自然なのでは？」なんて言っていたりもしますが、「乱立から集約へ」というAndrewの意見と「乱立から不可視へ」という私の意見は異なる点もありますが、共通UXとして自然に溶け込んでいくことにならないと普及しない、という点については一致していると思います。

雑にGoogle翻訳したものを貼っておきます。

インフラストラクチャとしてのアイデンティティウォレット デジタル ID ウォレットの世界は、これからさらに面白くなりそうです。欧州連合はEIDAS v2を展開し、モバイル運転免許証の採用は米国全土で加速しています (最近の例としては、ニューヨークとカリフォルニア)。そして、これらすべてをサポートする重要な標準 (とりわけ、ISO 18013-5や検証可能な資格情報など) はますます確立されつつあります。今後 3 ～ 5 年以内に、インターネット ユーザーの大半が少なくとも一部の資格情報をデジタル ID ウォレット (以下、単に「ウォレット」) に保存するようになることはほぼ間違いないでしょう。
このような普及により、これらのウォレットをエンドユーザーのアプリケーションやサービスとして考えるのをやめ、「インフラストラクチャ」として考え始める時期が来ています。
ウォレットの再考: サービスからインフラへ オンライン サービス (およびアプリ) は本質的に競争的です。個人用タスク管理システムを例に挙げてみましょう。市場には数多くのシステムがあります。OmniFocus、Amazing Marvin、Remember the Milk、Todoist、Toodleoo などです。中には、Apple の世界の「リマインダー」のように、オペレーティング システムやメーカーのエコシステムに組み込まれているものもあります。
それはそれで問題ありません。基本的な機能はほぼ同じです (タスクの作成、タスクの完了チェック)。ただし、システムによって提供される機能は異なり、それが自分にとって役立つかどうか、また、お金を払いたいと思うかどうかはわかりません。タスクのタグ付けなどの一部の機能は、1 社または 2 社のベンダーの USP として始まりましたが、需要が高まり、今ではすべてのタスク管理ツールの必須機能となっています。
ウォレットが他のアプリと異なる理由 では、なぜウォレットが違う必要があるのか​​、と疑問に思うかもしれません。おそらく、物理的な財布と同じように、誰もが欲しがるわけではない機能をウォレットに求めることになるでしょう。例: 私は仕事で年に数回米国に行きます。米国は英国よりもはるかに現金中心の経済です。英国では、今では現金を持ち歩くことはほとんどありません (自転車に乗っているときは別ですが、そのときは緊急時用に 20 ポンド紙幣を持っています)。米国では持ち歩きます。米ドル紙幣は英ポンド紙幣よりも長いので困ります。そのため、私の財布には、英国で販売されている多くの財布とは異なる寸法が必要です。言い換えれば、物理的な財布には米ドルと GDP の両方をネイティブでサポートする必要があるのです。
デジタル ID ウォレットの限界 では、デジタル ID ウォレットとの違いは何でしょうか?
簡単です。もし物理的な財布が、使用したい通貨をネイティブにサポートしていない場合は、回避策を簡単に実装できます。紙幣を別の方法で折りたたんで、収まるようにすることができます。もちろん完璧ではありませんが、うまくいきます。
しかし、これは私のデジタル ID ウォレットには当てはまりません。たとえば、ある国家が、自国の認証情報に有効なウォレットは特別な国家ウォレットのみであると決定し、そのウォレットが他のすべての人が頼りにしている検証可能な認証情報の一部の機能をサポートしていない場合 (または、更新が十分に速くない場合など)、私は困ってしまいます。私の唯一の選択肢は 2 つの別々のウォレットを実行することですが、その状況は急速に悪化する可能性があります。
「でも、これは先進国の国際的なジェットセッターが抱える問題のように思える」とあなたは言うでしょう。確かに、他の機関が同様の道を歩み始めたと想像してみてください。スーパーマーケットのポイント制度に参加するのですか？ ウォレットが必要です！ 銀行口座を開設するのですか？ ウォレットが必要です！ 学歴、専門資格、または福利厚生の資格が必要ですか？ はい、ウォレットがさらに必要になります。特定の資格情報がどのウォレットに入っているか思い出せなくなるのも時間の問題です。また、デバイスを紛失した場合（紛失した場合）、またはアップグレード時に使い捨てウォレットの一部を転送し忘れた場合の資格情報回復プロセスは、考えたくもありません。ウォレットの急増は採用を妨げるでしょう。
インフラとしてのウォレットの力 企業もソフトウェアベンダーも、ウォレットを機能として考えるのをやめる必要があります。ウォレットは実際にはインフラストラクチャ コンポーネントです。この文脈で「インフラストラクチャ」とはどういう意味でしょうか。鉄道や電力網を考えてみてください。少なくとも、それらが何を行うか、どのように機能するかという基本的な点については、誰もが同意しています。それらは大規模で、(文脈上) 広く利用可能です。そして、本当の価値は鉄道や電力網自体からではなく、それらの上に構築できるサービスから生まれます。言い換えると、それらは本質的に一貫性があり、相互運用性があり、遍在的で、基礎的なものです。または、Webster の定義によれば、「下部構造または基礎となる基盤。特に、コミュニティ、国家などの継続と成長が依存する基本的な設備と施設」です。
では、ウォレットについて考えてみましょう。ウォレットが利用可能になる可能性が最も高い最終段階は、ほとんどの人 (消費者、従業員、市民など) が携帯電話プロバイダーからウォレットを入手することだと私は考えています。彼らは、入手するウォレットに基づいてプロバイダーを選択することはなく、そのプロバイダーのデフォルトのウォレットを単に使用します。なぜなら、彼らはウォレット自体の機能にはあまり関心がないからです。彼らは、ウォレットが使いやすく、信頼性が高く、安全に動作し、広く受け入れられ、それを使用してさまざまなデジタル、物理、ハイブリッド サービスにアクセスできることを望んでいるだけです。
この結果は、実は私たち全員にとっての利益です。個人のデジタル ID は、使いやすさ、アクセシビリティ、インクルージョン、顧客維持、セキュリティ、プライバシーなど、デジタル環境のさまざまな領域に革命をもたらします。その結果、企業や、地方レベルと国家レベルの公共サービスを含むその他の大規模組織に、顧客獲得/維持の向上、セキュリティとプライバシーの体制の改善、コスト削減などのメリットがもたらされます。
アイデンティティのための新しいアーキテクチャ さらに、ウォレットをデジタル ID インフラストラクチャの一部として考えると、興味深く重要な新しいアーキテクチャがいくつか生まれます。ウォレットはシグナルを提供できます。ウォレットは「カウンセラー」になることができます。当社のエンタープライズ展開では、ウォレットからの入力を積極的に取得したり、それに応答したりできます。ウォレットは継続的な ID ランドスケープの一部になります。
好むと好まざるとにかかわらず、ウォレットはインフラストラクチャです。そのインフラストラクチャをできるだけシンプルで、誰にとっても便利なものにしましょう。

It is three times the price to rent a car in the center of Auckland for the day than on Waiheke Island - that does not compute.

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Microsoft Recall to be unremovable, another “AI talent raid” by Amazon, Ticketmaster struggles to handle common load challenges – sometimes, rise and fall of OpenSea, and more.

More trad tech companies to move off public cloud? In a recent report, 83% of chief information officers say they want to move more workload to private clouds. Cloud computing has matured and is becoming more expensive, while private cloud technology is increasingly accessible.

CockroachDB: open source product goes proprietary. Another VC-funded open source product closes itself off, in hope of generating more revenue and profits. Many users need to open their wallets and buy annual licenses, or fork and maintain an older version of CockroachDB.

Software glitch almost floods Amsterdam. Dams near the Dutch capital that were supposed to be closed, mysteriously opened at night during a violent storm. Disaster was averted by a curious employee – and the workaround to avoid similar incidents includes 24-hour human monitoring of dams and water levels.

1. Industry pulse Microsoft Recall to be unremovable

Microsoft’s struggle with its controversial Recall feature continues. Recall is a “photographic memory” feature planned for Windows 11 which takes constant screenshots of a device, and makes it searchable. Its launch in June was a disaster, as Microsoft skipped basic security and privacy approaches – making these screenshots easily accessible for bad actors. A week after, Microsoft “recalled” Recall, confirming that it will not ship it with these glaring holes.

Now, the Redmond-headquartered tech giant is doubling down on Recall: it will be shipping on Windows 11, and the feature will have no option to be uninstalled. Users won’t be able to disable it.

The problem with making Recall core to Windows with no way to remove it or turn it off at the OS level, is that it could become a dealbreaker for enterprise clients which will not want an invasive, potentially insecure feature to run on machines by default. It’s also strange to see Microsoft so set on shipping this feature that is clearly designed to make some use of the hardware GPUs, even though the utility is still questionable. The company will surely keep getting bad press, and privacy-focused techies will have one more reason to stay clear of Windows.

Another “AI talent heist”

Read more

“Communities that want to build comprehensive public calendars will be able to do so using a hybrid approach that blends existing iCalendar feeds with feeds synthesized from web calendars. It’s not a perfect solution, but with LLM assistance it’s a workable one. And who knows, maybe if people see what’s possible when information silos converge, the common tools that can ease convergence will seem more attractive.” — An LLM-Turbocharged Community Calendar Reboot

Part of the LLM series at The New Stack.

[Elahe Izadi at The Washington Post]

I adore The 19th. It's making big moves, and that's good news for everyone.

"What [Emily] Ramshaw and fellow co-founder Amanda Zamora started in January 2020 — a newsroom with just one reporter and no website — has grown into a digital operation that has raised nearly $60 million and employs 55 people. And in a sign of its growing ambitions, the 19th has now hired veteran news executive LaSharah Bunting, CEO of the Online News Association, as its first vice president, a role created to build up the 19th’s fundraising and budget operations."

It's also grown an endowment, which allows it to have a safety net and continue to grow and experiment. The ambition for the endowment to underwrite the newsroom's operations is meaningful: this would represent a fund designed to allow reporting on gender, politics, and policy to be undertaken sustainably. I don't know of any other similar fund in media.

Not mentioned here but extremely relevant: the amazing work Alexandra Smith, its Chief Strategy Officer, has been doing to redefine how to think about audience and reach on a fragmented web.

These are all signs of a forward-thinking newsroom that isn't content to simply accept the status quo - and, crucially, plans to stick around.

#Media

[Link]

Yesterday the Internet Archive lost its appeal in the digital lending case it’s been fighting for the last few years.

In March 2020, the Internet Archive, a San Francisco-based nonprofit, launched a program called the National Emergency Library, or NEL. Library closures caused by the pandemic had left students, researchers, and readers unable to access millions of books, and the Internet Archive has said it was responding to calls from regular people and other librarians to help those at home get access to the books they needed.

It was a useful program, and the archival has merit, but publishers argued that the Archive overstepped, and the courts eventually agreed.

Regardless of the merits of the case, I believe the Internet Archive is an obvious public good, and an outcome like this has the potential to do it real harm. This opinion led me to post an offhand comment on Threads:

People who follow me tend to also be at the intersection of tech and media, so I figured extra context wasn’t needed. They were on it. And I figured that anyone who wasn’t clued in probably didn’t care and could just keep scrolling.

Which, uh, is not how it went down.

The Threads algorithm apparently surfaced my post in the feeds of a bunch of other people with a wholly different set of interests, who were — inexplicably to me — incredibly angry that I hadn’t provided any further context.

A whole bunch of people apparently forgot they can, you know, just Google something:

But the comments that really surprised me were the ones that accused me of engagement farming. I’ve never received these before, and it made me wonder about the underlying assumptions. Why would this be engagement farming? Why would someone do this? Why would they assume that about me?

It might have something to do with Meta’s creators program, which pays people to post on the platform. The idea is that popular influencers will lure more users to the platform and it can therefore grow more quickly.

The amounts are not small: a single popular post can earn as much as $5,000. It’s an invite-only program that I am not a part of; it looks like you need to be an existing Instagram influencer to be asked. While I’m a lot of things, that is very far from being one of them.

Because the program is not available to all, and because it’s unlabeled, it’s not clear who is a part of it and who isn’t. So anyone could be trying to farm engagement in order to make some extra money. And because anyone could be, it becomes the default assumption for a lot of people. If you had the opportunity to make an extra $5,000 for a social media post, why wouldn’t you? And as a result, trust in peoples’ underlying motivations has disintegrated. Everyone must be just trying to get as many views on their posts as possible.

Over time, this has the potential to become pernicious, eroding trust in everything. If X has fake news, Threads is assumed to have fake views: engagement by any means necessary.

To be clear, if I was a part of the program — which, again, I’m not — I wouldn’t do anything differently, except to clearly announce that I was part of the program. I’m not an entertainer, an influencer, or a public figure. Like most of us, I’m just some person posting offhand thoughts into a social media app; anything else feels, honestly, disingenuous and like far too much work. But now I understand how fast trust has eroded, I wonder if the ability to build authentic communities on the platform is hanging on by a thread.

Someone’s Having A Larf …

It’s the billion-dollar baby of artificial intelligence. Safe Superintelligence Inc., the AI startup launched by former OpenAI chief scientist Ilya Sutskever, announced Wednesday it had raised a whopping $1 billion from big-name venture capital firms, including Sequoia and Andreessen Horowitz.
That’s a huge amount of money for a three-month-old company with 10 staffers, no product and a “singular focus” on creating “safe” AI rather than a money-making product (at least in the short term).

💬 The Information

Shout Outs To @dsearls@journa.host and @dave in the 🔗 latest entry from ‘Jobsworth’

BTW - while I am here 🔗JP published another great post the other day about Cricket - specifically Cricket stats and records.

He opened with:

You have been warned. This one’s for hardcore cricket nuts. Red-ball nuts. Five-day nuts. The hardest of the hardcore. No coloured pajamas here.

I am definitely not a hardcore cricket nut - and still read the entire thing to the end. Fascinating.

Qwen2-VL: To See the World More Clearly

Qwen is Alibaba Cloud's organization training LLMs. Their latest model is Qwen2-VL - a vision LLM - and it's getting some really positive buzz. Here's a r/LocalLLaMA thread about the model.

The original Qwen models were licensed under their custom Tongyi Qianwen license, but starting with Qwen2 on June 7th 2024 they switched to Apache 2.0, at least for their smaller models:

While Qwen2-72B as well as its instruction-tuned models still uses the original Qianwen License, all other models, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, and Qwen2-57B-A14B, turn to adopt Apache 2.0

Here's where things get odd: shortly before I first published this post the Qwen GitHub organization, and their GitHub pages hosted blog, both disappeared and returned 404s pages. I asked on Twitter but nobody seems to know what's happened to them.

Update: this was accidental and was resolved on 5th September.

The Qwen Hugging Face page is still up - it's just the GitHub organization that has mysteriously vanished.

Inspired by Dylan Freedman I tried the model using GanymedeNil/Qwen2-VL-7B on Hugging Face Spaces, and found that it was exceptionally good at extracting text from unruly handwriting:

The model apparently runs great on NVIDIA GPUs, and very slowly using the MPS PyTorch backend on Apple Silicon. Qwen previously released MLX builds of their non-vision Qwen2 models, so hopefully there will be an Apple Silicon optimized MLX model for Qwen2-VL soon as well.

Tags: vision-llms, llms, ai, generative-ai, qwen

Me:

What about tomorrow?

Reply:

Would be good except that I will out of town celebrating my first wife’s, second husband’s, 70th. Should be a good bash.

.. now that’s what I call an extended family that truly embraces the concept.**

📺 Kleo

Each episode kept me coming back for more. An increasingly difficult thing to do in theses ‘streaming times’.

Kleo on 🔗 Reelgood

’All’ My TV Show Reviews

 

No Words.

📼 Just Watch

Me last July … 🔗 Read

Me last August … 🔗 Read

Why?

Because as the scuttle around that movie at the time suggested - nothing was as it seemed.

🔗 Tim Ballard’s Claims to Fight Sex Trafficking Made Him a MAGA Star. These Women Told Police He Abused Them.

こんにちは、富士榮です。
4月にこちらでも書いたVittorio Bertocciアワードですが受賞者が決まったみたいです。
Celebrating Excellence: Meet the first Vittorio Bertocci Award Winners! https://digitalidadvancement.org/news/celebrating-excellence-meet-the-first-vittorio-bertocci-award-winners/ 相変わらずのこの写真(笑)。愉快なヤツでした


おさらいですがこのアワードは故Vittorio Bertocciの功績を継承すべくDIAF（Digital Identity Advanced Foundation）が設定しているアワードです。
今回受賞が決まったのはTrack1と2の受賞者でTrack1の受賞者の方は10月末のInternet Identity Workshop（IIW）でお会いできそうです。
受賞したのは以下の方々 Erick Domingues（Track 1） ブラジルでRaidiamのプログラム・マネージャーをやっている人 FAPIの実装とかやっている方みたいですね Frederico Schardong（Track 2） ブラジルのサンタ・カタリナ大学でコンピューターサイエンスの研究をしている人 デジタルアイデンティティ、自己主権型アイデンティティなどの研究をしているみたいです Jen Schreiber（Track 2） Women Who Codeでエリアディレクターをやっている人 デジタルアイデンティティの実装を大規模サービスでやっている人っぽい
日本人も応募すればいいのになぁ。。

🔗 Raw data: Book and newspaper consumption - Kevin Drum

There is a mantra in the world that people ‘don’t read’ (anymore?).

And on top of that - if they do - (very) short form - definitely not books.

I have long thought this to be wrong.

They certainly might not read your stuff. Or my stuff, but they do read - BUT - they read what interests them , what engages them and definitely not limited to stuff that appears at the top of lists.

Kevin’s short analysis on the data says that today we are spending twice what we spent on physical books 40 (or so) years ago. As he says - buying is not reading (👋🏼 to the Unlibraries) .. but beyond physical books there are ebooks, blogs, newsletters … yup - we definitely aren’t reading any more - except when we do.

At Bar del Pla in Barcelona

こんにちは、富士榮です。

引き続きSIDI Hubのケープタウン会合のレポートをよんでいきましょう。

今回はガバナンスです。

Governance - 報告者：Elizabeth Garber

ガバナンスに関してはGail Hodgesが担当しました。SIDI Hubの内部の各活動をローカルガバメントと連携・調整しつつ、それぞれのワークストリームへマッピングしていく、ということをやっています。

どうしても国際的な枠組みで動こうとするとこのような整理をしないとごちゃごちゃになっちゃうんでしょうね。

その他にも、

ユースケースの優先順位づけへのローカルの意見を反映しやすい仕組みの必要性 国ごとの個別のデータフィールドを除けば技術的なアラインメントはほぼほぼ行けそう エコシステムのガバナンスも考えないといけない アーキテクチャ上の決定はローカライズされるべきである（特にオープンソースにすべきところとローカル単位でクローズにするかどうかを決められる部分の決定ができるようにするなど） 既存のImplementationが相互運用性があるのか確認した方がよいのでは など、持続可能な形で運用していくための意見だしが行われたようです。

あとは、ガバナンスやオペレーションを設計する上で考慮すべき事項についても話あわれたようです。

マルチ言語を含むスキーマの設計（どうしても英語がプライマリになってしまいがち） Relying Partyの登録 Issuing AuthorityのDiscovery vLEI的（そのものとは言っていない）な暗号学的に検証可能な法人エンティティを識別するための情報 x.509とOpenID Federationの違いなどの技術的な相違点 OSSのコードと認証の仕組み OECD、G20などとのMOUの話 などなど。やはりグローバルで相互運用性を担保しつつ持続可能にするには考えるべきことはいっぱいです。一歩一歩整理をつけていっている途中ですがGailお疲れ様です・・・

ということで次回は最低限の要求事項のまとめのセッションです。

こんにちは、富士榮です。

引き続きSIDI Hubケープタウン会合のイベントレポートを見ていきます。今回はトラストフレームワークのセッションに関するレポートです。

早速見ていきましょう。

Trust Framework - 報告者：Stephanie de Labriolle 次はトラストフレームワークのワークストリームです。このセッションはElizabeth GarberとOpen Identity ExchangeのNick Mothershaw（リモート参加）が担当しています。
Stephanieのレポートによると参加者の半数くらいしかトラストフレームワークについて馴染みがなかったようですが、各国の法律やルールなどはすでにトラストフレームワークの主要な要素を持っているためElizabethからその辺りは説明がされたようです。
OIXのこの辺りの資料で説明したとのことです。
セッション中ではトラストフレームワークの利点の例として以下が挙げられたとのことです。
As a state, I want to flawlessly recognize an individual and know they are unique so that I can offer the right access and services a Trust Framework defines requirements for Identity Proofing and Levels of Assurance 国家としては、個人を完璧に認識し、その人が一意であることを知って、適切なアクセスやサービスを提供できるようにしたい。 トラストフレームワークは、アイデンティティの証明と保証レベルの要件を定義する。 As a user, I want to know that my private information is safe so that I can avoid scams, identity theft, and harms in the digital and physical worlds a Trust Framework defines requirements for Privacy, Security, Relying Parties/Verifier Obligations, Data Management, etc. ユーザーとして、私は自分の個人情報が安全であることを知りたい。そうすれば、詐欺、個人情報の盗難、デジタルおよび物理的な世界での危害を避けることができる。 トラストフレームワークは、プライバシー、セキュリティ、依拠当事者／検証者の義務、データ管理などの要件を定義する。 As an Identity Issuer I want to know that the information is going to a trustworthy place so that I can protect users’ data a Trust Framework defines requirements for Relying Party/Verifier Obligations, and Trust Registry protocols ID 発行者として、情報が信頼できる場所に送られることを知り、ユーザのデータを保護したい。 トラスト・フレームワークは、依拠当事者／検証者の義務、およびトラスト・レジストリのプロトコルの要件を定義する。 As a user, I want to know that I can safely use my credential anywhere to prove who I am, what I can do, and to access resources the Trust Framework defines requirements for Credential Standards ユーザとして、自分が誰であるか、何ができるかを証明し、リソースにアクセスするために、自分のクレデンシャルをどこででも安全に使用できることを知りたい。 トラスト・フレームワークは、クレデンシャル標準の要件を定義する。
そして、Open Identity Exchange（OIX）は以下の8つのトラストフレームワークの分析を実施してきました。
日本は？？？ 安心してください。その後OpenIDファウンデーションジャパンの有志でちゃんと進めてくれています。次の日本会合ではその辺りも発表があると思います。
なお、OIXの分析の結果、OIXが「デジタルIDトラストフレームワークのDNA」として定義している「一般的なポリシー・ルール」と「アイデンティティ保証に関するポリシー」の2つの主要テーマと、関連するサブテーマが設定されています。
ここでも小グループに分かれてディスカッションを行い、トラストフレームワークの要素を用いて各国の状況の分析を行っています。 詳細はレポートを見ていただければと思いますが、こんな感じで分析したようです。

また、トラストフレームワークのベネフィットについても議論が行われました。 結果、以下のようなまとめが行われたとのことです。 この分析のベネフィット 相互運用性の推進 データ保護とセキュリティの促進 コストの削減 包摂 デジタル経済の発展 官民サービスの提供への志向 一方でチャレンジとして以下も挙げられています。 ポリシーをどのように運用していくか デジタル化をどのように進めるか 人的要因・サイロ化・リーダーシップの課題への対応 インフラの不足 スキル不足 地理的な問題、規模、セキュリティ上の問題 そして、何が足りないのか？についても議論が行われ、「専門家のアドバイスの中立性をどのように担保・確認するのか？」などについても語られたようです。どうやら一部の国の政府は外部有識者へのアドバイスを求める際、公平性や中立性に課題がある、と考えるケースもあるようです。

なお、トラストフレームワークの議論についてもUNHCRとの関連で議論が行われました。

Meanwhile, non-jurisdictions focused on how to support UNHCR, which has the challenge of serving 130M current refugees that are part of the UNHCR system and under their protection. They have integrations with about 7 strategic partners including refugee origin and destination countries, and they need to have 50-60 more integrations to national civil registry systems. Some users entering the system will have documents from their origin country and the origin country system of record may be accessible to check data against and people may have mobile devices (e.g. Ukraine), while other individuals may not have mobile devices and may be stateless or originate from failed states where records are not available.

一方、非管轄当局は、UNHCRをどのように支援するかに焦点を当てた。UNHCRは、UNHCRのシステムの一部であり、その保護の下にある1億3,000万人の現在の難民にサービスを提供するという課題を抱えている。UNHCRは、難民の出身国や目的地を含む約7の戦略的パートナーと統合しており、さらに50～60の国別市民登録システムとの統合が必要である。システムに入る利用者の中には、出身国の文書を持っていて、データを照合するために出身国の記録システムにアクセスできたり、携帯端末を持っている人（ウクライナなど）がいる一方で、携帯端末を持っておらず、無国籍であったり、記録が利用できない破綻国家出身であったりする人もいる。 

数多くの難民を支援するUNHCRでは多くの国のシステムとの相互運用性を実現する必要がありそうです。しかしながら難民の状況はさまざまなので国民IDシステムへのアクセスができない場合などもあるので非常に難しい舵取りが求められている状態のようです。

またウクライナの状況を例としてUNHCRに何が求められるのか？についても議論が行われ、難民登録やデジタル技術による解決なども話題に上ったようです。

ということでトラストフレームワークのセッションも終わりです。

次回はガバナンスです。

 

💬

💬

💬

I have been wanting to experiment with creating some code performing retrieval augmented generation (RAG) for an LLM. So far I am 0 for 2. In both examples I tried, that were professed to work, it fell far short. I blame embedding and retrieval. I also blame myself… who else is there to blame. I know RAG works in some form because I can go onto ChatGPT upload a PDF and prompt about it with expected and quality results; but, when trying to replicate similar outcomes from code and the OpenAI API I get disappointing results. Here’s the story.

Experiment one was part of a Udemy course on RAG and mastering some better approaches than naive RAG. Specifically RAG with query expansion. It was a very short course but had a couple of cohesive Python examples that built out over the course material. It included the document to use for the augmentation. Additionally, unlike experiment two, it largely didn’t rely on a ton of abstraction of the underlying basic concepts. After reading and parsing the text from the PDF I used RecursiveCharacterTextSplitter and
SentenceTransformersTokenTextSplitter from Langchain to create my chunks. Added my chunks to Chroma db in memory using their default embedder. Took my query and performed the retrieval of 5 chunks from the Chroma db. No reranking performed. The returned chunks were at best partial results and at worst just not as good as expected. What I hadn’t noticed about the code from the instruction when I went through the class is that it never passed the chunks back as context to the LLM the second time along with the query to get an answer. Kind of an important part to be missing. I can tell from the returned chunks it would not have produced a satisfactory answer had the closing action before performed. I tried with differing chunk sizes and overlaps and never received better results. I tried with my own document and faired no better. I chalked it up to a disappointing and poor example. Perhaps this is why it didn’t go the final step to pass it all back to the LLM for a response.

I moved on to a second experiment that used a bit more abstraction by relying on Langchain significantly more. It was also doing naive RAG, not augmenting the prompt from the LLM initially in any way. This time it did have the pass to the LLM in the end to get the LLM response. Looking at the chunks it retrieved I could tell I wasn’t going to get a satisfactory response. I had asked it for a list of 10 principles that were specifically outlined in the document in a single paragraph. The best in all my attempts was that I got 2, maybe 3 of the list. Again I played around with chunk size and overlap and generally only got worse results. The results in the demonstration had not seemed much better when I evaluated it more closely.

All in all I need a third+ experiment. I am trying to get a response even remotely similar to what I get from the ChatGPT. Obviously, my examples are not high enough quality.

Do you have such an example?

Time to ask ChatGPT, Claude, Github Copliot, or Cursor — sounds like a job for Cursor — to create a python program for me and see how it functions.

Q: “I’d like to make a better case for paying down tech debt on my team. What are some proven approaches for this?”

The tension in finding the right balance between shipping features and paying down accumulated tech debt is as old as software engineering. There’s no one answer on how best to reduce tech debt, and opinion is divided about whether zero tech debt is even a good thing to aim for. But approaches for doing it exist which work well for most teams.

To tackle this eternal topic, I turned to industry veteran Lou Franco, who’s been in the software business for over 30 years as an engineer, EM, and executive. He’s also worked at four startups and the companies that later acquired them; most recently Atlassian as a Principal Engineer on the Trello iOS app. 

He’s currently an operating partner at private equity firm SilverTree Equity, and technical co-founder at a new startup. Lou says he isn’t delighted about the volume of tech debt accumulated during his career, but is satisfied with how much of it he’s managed to pay down. 

In this guest post, Lou shares insights about how to approach tech debt. Later this year, he’s publishing a book on the subject. For updates on this upcoming release, subscribe here.

In this article, we cover:

Pay down tech debt to go faster, now. It’s common for less experienced engineering leaders to assume that focusing on features and ignoring tech debt is how to move faster. Lou used to agree, but not any more.

Use tech debt to boost productivity. Three examples of when tackling tech debt empowered engineering teams to move faster, right away.

Link tech debt and value delivery. When tech debt is tackled sensibly, it delivers business value. At Trello and Atalasoft, this was how Lou got engineers and management to appreciate the task.

Make tech debt’s effects visible. Dashboards are an effective way to visualize the impact of tech debt. A few examples.

Factor in time saved when thinking about productivity. Reducing tech debt typically improves coding, reviewing, and deployment for everyone.

With that, it’s over to Lou:

1. Pay down tech debt to go faster immediately

What is tech debt?

I define tech debt as any problem in the codebase that affects programmers by making it harder to make necessary changes. As a programmer, I wanted to fix such issues because they slowed me down. But as a manager, I had to ensure the team delivered value to stakeholders. I’ve gone back and forth between these roles during my career, and made mistakes in both directions – but I also learned a lot about getting the balance right.

Reducing tech debt pays off immediately in faster builds

In 2010, I was head of development at Atalasoft, a company in the .NET developer tools space. I was obsessed with shipping, and spent all my time worrying about delivering the features in our roadmap. Over time, we improved at this, which showed up in our revenue growth and eventually led to an acquisition. 

We were in a competitive market with more opportunities than we could handle. We had fewer than ten developers, but we were bootstrapped, so had to stay profitable and couldn’t just hire ahead of revenue.

The stakes got even higher after we were acquired. We had two years to deliver on an ambitious roadmap, for which there was an earnout bonus contingent upon delivery. If we didn’t deliver, we’d likely be classified as a failed acquisition. Our new owners had just had one such failure, which led to downsizing and an office closure.

My mindset was that any time spent on technical debt meant we’d fail to deliver on the roadmap. We couldn’t risk the deadline by wasting time cleaning up old messes, and had to choose between tech debt and roadmap delivery. In hindsight, I see this was wrong. I learned that the mindset of ignoring tech debt hurt my team.

Following an exit interview in which a departing engineer mentioned tech debt had contributed to their decision to leave, I started asking people during 1:1s how they felt about it. Their responses showed their frustration with me. 

By then, I had been a developer for over fifteen years, and had worked in codebases with a lot of debt, so I knew what it was like. But by now, I was a manager who sometimes contributed code, but had forgotten what it was like to be thwarted by the codebase every day. To my team, I was part of the problem. They had been trying to tell me this, but I didn’t get it. Their departing colleague with nothing to lose in their exit interview finally got through to me and helped me understand the depth of the issue, and that it was slowing everyone down.

I learned an important lesson: the cost of tech debt is borne daily by your team, and you risk damaging motivation and raising attrition by ignoring it. Even if you have every reason to move forward without addressing tech debt, being an empathic manager requires you at least do something. Doing nothing – like I did – is not a good option.

So we started making changes. The biggest problems were with our build system and installer because they affected every developer and every product. It was a tangled bag of legacy code that needed constant maintenance, but it wasn’t very big, and I approved a plan to rewrite it with modern tools. It was a small experiment but paid off right away in quicker CI builds and an easier-to-modify codebase. Most importantly, I saw it didn’t derail our roadmap, so we took on other small initiatives.

This taught me another lesson about addressing technical debt. I had thought of it as something that might pay off in the long run. Might. This belief made it hard to justify doing it when I had to deliver on short-term goals. But instead, something else happened:

We paid off tech debt and increased productivity instantly! We had a build with faster feedback loops, less cognitive load, and which didn’t make developers frustrated when they had to add to it, which happened regularly. Updates were made with less code and without breaking things. It was an example of tech debt reduction paying off in increased developer productivity, right away.

Learning the cost of too much rewriting at Trello

I got my next lesson at Trello where I worked on the iOS app. The codebase was three years old when I joined in 2014. It had understandable tech debt because they needed to move fast, after going from 0 to 6 million sign ups. The devs working on it were founding engineers, working as described by The Pragmatic Engineer in Thriving as a Founding Engineer, and seeking product-market fit. Our biggest tech debt issue were some frameworks that made it fast to build a simple app, but held us back as the app got more complex.

Our own choices were influenced by the speed of Apple’s updates to iOS. The iOS 7 update completely changed the iOS design language and its network APIs. Later, iOS 8 introduced presentation controllers that gave developers much control over the animation when new views are shown. Unfortunately, the iOS 8 change broke our navigation code and caused crashes. These added up and started to make our code seem antiquated.

Left: how most iOS 6 apps looked like. Right: the iOS 7 version. iOS 7 introduced a ‘flatter’ look, and required lots of code changes to implement this new UI. Screenshot is from my app called Habits

Our code got even more complex when Apple decided to feature Trello on physical iPhones at Apple Stores. To be in stores, we needed a build that worked without an account or a network, so a mock backend was embedded in it for demo purposes. We didn’t want to maintain a separate codebase, so had a lot of random bits of demo-mode logic that stayed for years. 

At Trello, I was coding every day and all this was in my face. Luckily, we were a small team of three developers, so my direct manager was also coding every day and was empathetic to the problems.

We did rewrites as we went, but sometimes went too far. To deal with the presentation controller problem of iOS 8, we developed a new paradigm for screen navigation inside the app, and rewrote all navigation to use it. This approach was the exact opposite of what I did at Atalasoft, where I’d ignored all tech debt. 

Unfortunately, the approach of rewriting early turned out to be overkill. In hindsight, we could have just corrected the places that had crashed, and then lived with the code we had. Instead, we spent a few months designing and implementing a new, non-standard way of writing navigation code, but forgot a vital lesson that one of our founders, Joel Spolsky, identified in 2000 in Things You Should Never Do:

“We’re programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.

There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:

It’s harder to read code than to write it.”

On the Trello engineering team, we were all very familiar with this article and quoted it to each other often, but it still sometimes bit us. The urge to rewrite a system instead of fixing it is strong, and we couldn’t resist! We should have addressed the few complex navigation cases that crashed our code without the full rewrite.

Size tech debt payment to be proportional to value. This is the biggest lesson I learned on this project.

I’ve seen both extremes of dealing with tech debt:

As a manager, I was overly resistant to devoting time to dealing with technical debt

As an engineer, I was exposed to its problems every day and didn’t resist the urge to pay it off enough 

These two extremes form the fundamental tension of dealing with tech debt. As usual, there needs to be a balance, but finding it is not so easy. 

The heuristic I use to pay tech debt these days is this: by reducing a specific tech debt, can I increase developer productivity and deliver business value right now?

If I can’t, then I don’t pay it down. 

When the debt is so big that it couldn’t possibly deliver value now, or the value is invisible so nobody sees it, I do something else. Let me break down my heuristic…

2. Use tech debt to increase productivity

I try to pay down a little bit of tech debt regularly by making small cleanup commits as I go. I started doing this more intentionally after reading Kent Beck’s book, Extreme Programming Explained, in 1999, which introduced me to automated unit tests and continuous integration. Then, when I read Martin Fowler’s Refactoring, I started to see how to improve a codebase over time with very small, behavior-preserving changes checked by unit tests. In both books, and in others like Working Effectively with Legacy Code by Michael Feathers, and Kent Beck’s recent, Tidy First?, the authors stress that technical debt is inevitable, and that the main way to curtail it is to be constantly fixing it with small improvements enabled by unit tests and mechanical refactoring. I agree.

Unit tests, refactoring, and continuous integration are ubiquitous in the kinds of software I write, which are B2B SaaS productivity applications. Even making small improvements on an ongoing basis is common among my coworkers. It doesn’t take long, and there are usually quick wins to be had, like making the code more readable, or using a unit test to show how the code is supposed to work. Even in frontend code, Trello iOS adopted Model-View-ViewModel (MVVM) so we could test view-logic. We got the immediate productivity benefit of being able to run view code repeatedly without needing to manipulate a running app through several screens to check that our changes worked.

The issue is when the debt is large, which is where I struggled. My problem at Atalasoft was not with small improvements; it was with the bigger ones where I’d have to trade off current benefits like delivering features for the roadmap, for uncertain future benefits.

But I realized something.

You can get productivity benefits immediately, even with larger initiatives. If you do it right, you will deliver feature work faster and with higher quality. In fact, I view tech debt proposals that don’t deliver instant developer productivity gains as suspect. 

Rewriting the build and installer at Atalasoft brought an immediate productivity boost. We had a backlog of problems and new additions, but the rewrite took one developer around a month, and when it was done many of the problems just went away because the new system was based on a framework wherein many problems could not occur, meaning we could close a bunch of reported bugs. The new system was unit testable, so we didn’t need to build and install the entire system during development to test our new changes while we were writing it. We also got more benefits later, but the instant benefits justified it.

At Trello, adding unit tests to a codebase helped me finish a project faster. When I joined in 2014, we were just about to start our internationalization (i18n) project, which I took on for the iOS app. One part was to write support for i18n-safe string interpolation (inserting variables or expressions into a string,) and pluralization (adjusting words to plural forms based on the number they refer to, to make the phrases grammatically correct) – which was only partially supported in iOS at the time. It’s standard string manipulation at its core, but in 2014 our iOS app didn’t have unit tests.

Without unit tests, if I had wanted to run the code, I’d need to run the app and then tap-tap-tap until I got to a specific string. I would have to do this for each kind of string I generated. But with unit-tests, I could just list all the examples with their expected results, and run tests in less than a second. So, I proposed to the team to add unit tests to our build and CI.

No one was against unit tests, but it hadn’t been a priority. Most of the code was UI or network code, for which unit tests are harder to write. But the code I was writing was highly testable, and in fact, it’s harder to write without tests. So, I added the unit test project to our workspace and wrote the string code. With the unit test project there, the other developers added tests to their work. I was there for six more years, and saw the benefits of the tests over time, especially in complex code like our sync engine. But that’s not why I did it: I added the unit tests to go faster immediately.

Also at Trello, creating an abstraction layer for the design system made us more productive. Eventually, we created a design system with a reduced set of fonts, colors, and other design attributes and specific rules for using them. Before, it was common to see hardcoded values in view controllers throughout the app, as each screen implemented the designer’s specification for that screen, which wasn’t always consistent. We could have just updated those lines to the new approved values, but it was the perfect time to make an abstraction for the design system itself. Doing this made it faster to write code that matched a design, and when a default in the design system changed, it would be reflected everywhere.

These three examples also adhere to another heuristic I use for finding the right balance with tech debt: coupling it with delivery of value.

3. Couple tech debt fixes with value delivery

Read more

history | tail -n 2000 | llm -s "Write aliases for my zshrc based on my terminal history. Only do this for most common features. Don't use any specific files or directories."

— anjor

Tags: llm, llms, ai, generative-ai

Schon morgen früh sollen die EU-Regierungen den umstrittenen Gesetzentwurf zur Chatkontrolle, der im Juni nach massiven Protesten von der Tagesordnung genommen wurde, jetzt doch unterstützen. Erreichen will die neue ungarische Ratspräsidentschaft dies nach einer Meldung des Nachrichtendienstes Contexte mit einer von Politico geleakten Detailänderung, nämlich dem Verzicht auf die besonders fehleranfällige Suche nach unbekanntem Material im Zuge der verpflichtenden Chatkontrolle (freiwillig soll sie möglich bleiben). Geplant bleibt aber ansonsten die automatisierte Durchsuchung und gegebenenfalls Ausleitung privater Chats, darunter auch Ende-zu-Ende-verschlüsselte Chats, die zuvor klassifizierte verdächtige Fotos oder Videos enthalten sollen. Lehnt ein Nutzer diese „Upload-Moderation“ seiner Chats ab, könnte er keinerlei Bilder, Videos oder URLs mehr senden oder empfangen. Signal und Threema haben angekündigt, dass sie ihre Dienste in der EU einstellen würden, wenn sie gezwungen würden, die vorgeschlagene automatisierte Massenüberwachung (sogenanntes „Client-Side Scanning“) umzusetzen.

Der ehemalige Europaabgeordnete der Piratenpartei, Patrick Breyer, ruft die Bürger jetzt dazu auf sich an die EU-Regierungen zu wenden: „Im Juni gab es unter massivem öffentlichen Druck noch eine hauchdünne Sperrminorität zur Rettung des Digitalen Briefgeheimnisses, aber kaum bemerkte Blitzaktionen, zwischenzeitliche Regierungswechsel und Minimalzugeständnisse können das schnell ändern. Falls die Chatkontrolle kommt, werden wir gängige sichere Messenger ganz einfach nicht mehr nutzen können – das bedeutet wir verlieren den Kontakt zu unseren Freunden und Kollegen in der ganzen Welt“, warnt Breyer. „Wollen wir wirklich, dass Europa weltweit führend beim Abhören unserer Smartphones und der flächendeckenden Überwachung der Chats von Millionen gesetzestreuer Bürger wird? Das Europäische Parlament ist überzeugt, dass diese orwellsche Chatkontrolle das dringliche Anliegen eines besseren Kinder- und Opferschutzes verrät, weil sie unweigerlich vor Gericht scheitern wird. Es fordert deshalb einen wirklich wirksamen Kinderschutz durch sicherere Apps, proaktive Säuberung des Internets und eine Pflicht zur Löschung illegaler Inhalte – nichts davon ist in dem dem neuesten Orban-Vorstoß vorgesehen, zu dem sich die Regierungen morgen positionieren sollen. Jetzt liegt es an uns ihn zu stoppen!“

Die Chatkontrolle auf vermeintlich „bekannte“ illegale Inhalte beschränken zu wollen, bezeichnet Breyer als Augenwischerei: „Egal mit welchem Ziel – auch die Post darf nicht einfach jeden Brief verdachtslos öffnen und durchschnüffeln. Gerade die von US-Konzernen schon heute freiwillig praktizierte Durchleuchtung nach vermeintlich bekannten Inhalten führt zur Ausleitung tausender völlig legaler privater Chats, zur Überlastung der Strafverfolger und zur massenhaften Kriminalisierung Minderjähriger. Massenüberwachung ohne jeden Anlass zerstört das digitale Briefgeheimnis und sichere Verschlüsselung, auf die wir alle und auch unsere Kinder dringend angewiesen sind.“

Breyers Infoportal zur Chatkontrolle: chatkontrolle.de

Hilf jetzt mit die Chatkontrolle zu stoppen

Python Developers Survey 2023 Results

The seventh annual Python survey is out. Here are the things that caught my eye or that I found surprising:

25% of survey respondents had been programming in Python for less than a year, and 33% had less than a year of professional experience.

37% of Python developers reported contributing to open-source projects last year - a new question for the survey. This is delightfully high!

6% of users are still using Python 2. The survey notes:

Almost half of Python 2 holdouts are under 21 years old and a third are students. Perhaps courses are still using Python 2?

In web frameworks, Flask and Django neck and neck at 33% each, but FastAPI is a close third at 29%! Starlette is at 6%, but that's an under-count because it's the basis for FastAPI.

The most popular library in "other framework and libraries" was BeautifulSoup with 31%, then Pillow 28%, then OpenCV-Python at 22% (wow!) and Pydantic at 22%. Tkinter had 17%. These numbers are all a surprise to me.

pytest scores 52% for unit testing, unittest from the standard library just 25%. I'm glad to see pytest so widely used, it's my favourite testing tool across any programming language.

The top cloud providers are AWS, then Google Cloud Platform, then Azure... but PythonAnywhere (11%) took fourth place just ahead of DigitalOcean (10%). And Alibaba Cloud is a new entrant in sixth place (after Heroku) with 4%. Heroku's ending of its free plan dropped them from 14% in 2021 to 7% now.

Linux and Windows equal at 55%, macOS is at 29%. This was one of many multiple-choice questions that could add up to more than 100%.

In databases, SQLite usage was trending down - 38% in 2021 to 34% for 2023, but still in second place behind PostgreSQL, stable at 43%.

The survey incorporates quotes from different Python experts responding to the numbers, it's worth reading through the whole thing.

Via PSF news

Tags: surveys, open-source, sqlite, python, pytest, postgresql, psf

こんにちは、富士榮です。

いよいよ来週に迫ったSIDI HubワシントンD.C.会合および来月の東京会合に向けて体（頭）を温めていきたいと思います。

前回は概要とまとめを見てきました。

https://idmlab.eidentity.jp/2024/09/sidi-hub.html

今回は各セッションを見てみたいと思います。

SIDI Summit Introduction - 報告者：Elizabeth Garber

まずはElizabethから、SIAのStephanieが担当した最初のセッションに関して報告されています。

SIDI is delighted to be at ID4Africa because the concept was born at an ID4Africa discussion about the development of national identity schemes with vastly different architectures – centralized, decentralized – and different models of governance. The SIDI organizers sought to promote discussions of interoperability between those systems to ensure that no nations are left behind as interoperability is established across others.

SIDI は ID4Africa に参加できることを喜ばしく思っている。というのも、このコンセプトは、ID4Africa のディスカッションで、中央集権型、地方分権型など、アーキテクチャが大きく異なり、統治モデルも異なる国家 ID スキームの開発について議論する中で生まれたからである。SIDI の主催者は、相互運用性が他国間で確立される中で、取り残される国がないようにするため、これらのシステム間の相互運用性の議論を促進しようとした。

ケープタウン開催ということもありID4Africaとの協業について触れられています。前回も触れましたが、アフリカという地域特性、統治モデル、国民IDのスキームの中でどのように相互運用性を実現するのかは非常に重要なことだと思われます。そしてこのテーマはUNDP（国連開発プログラム）の関心ごとの一つでもありますし、日本政府（外務省）からUNDPへのファンドが拠出されている分野でもあります。この辺りは日本からもっと支援ができると良いと思います。

また、同時にIntroductionセッションということもあり、SIDI Hubの戦略について紹介されています。

SIDI Hubの目的・何をして何をしないか、の紹介。

他にも5つのメインのワークストリームである、チャンピオンユースケースの探索、トラストフレームワークのマッピング、相互運用性のための最低限の要求事項、成功メトリクスの設定、ガバナンスについて紹介され、その中でもケープタウン会合のアジェンダでは以下の4つの点について重点が置かれているということについても紹介されています。

Identifying Champion Use Cases Preliminary Trust Framework Analysis of African models Minimum Technical Requirements for Interoperability Governance Approach + Metrics of Success (Two workstreams operating as one for now)
このセッションの最後でStephanieはSIDI HubのコンセプトについてSIDI Hubの戦略の中から以下のように締め括っています。

The concept is that we need a blueprint for how we build digital identity ecosystems within and across jurisdictions that will produce interoperable digital identity credentials by default. Achieving those outcomes will require measurement and metrics, policies, interoperable open standards, open source code (for many jurisdictions) and scientific analysis (e.g. liveness and presentation attack detection).

 このコンセプトは、デフォルトで相互運用可能なデジタル ID クレデンシャルを生成する、 管轄内および管轄をまたがるデジタル ID エコシステムを構築する方法の青写真が必要であるとい うものである。このような成果を達成するには、測定および測定基準、政策、相互運用可能なオ ープン・スタンダード、オープン・ソース・コード（多くの司法管轄権向け）、および科学的分析 （例えば、生存性およびプレゼンテーション攻撃の検出）が必要である。

Use Case Session Part 1 - 報告者：Mark Haine セッションの冒頭で語られた以下の宣言はさすが大陸文化って感じです。

"People need to move" was a clear opening statement. This is part of their nature. Often, this movement is across a border – a sentiment that resonated across representatives from the African continent, where many communities live and trade across borders. When life’s basic functions exist cross-border, people need ways to identify one another.

「人は動かなければならない」。これは彼らの性質の一部なのだ。多くのコミュニティが国境を越えて生活し、貿易を行っているアフリカ大陸の代表者たちは、この思いに共鳴した。生活の基本的な機能が国境を越えて存在する場合、人々は互いを識別する方法を必要とする。

前回のポストにも書きましたが国境とコミュニティの境目が異なる（オーバーラップする）のは地続きの大陸の特徴ですね。そして、その環境下でお互いを識別するための方法は非常に大切です。

The room mentioned ICAO and the passport standard, but the room agreed that there are many other use cases than travel across borders. In turn, identity solutions need to fit those use cases and there are many variations and different issues to consider. By looking at this we might find a working framework.

会場ではICAOとパスポート標準について言及されたが、国境を越えた旅行以外にも多くのユースケースがあることに同意した。そのため、ID ソリューションはそれらのユースケースに適合する必要があり、考慮すべき多くのバリエーションやさまざまな問題がある。これを検討することで、作業フレームワークが見つかるかもしれない。

 ちょっと興味深いですね。国境を超えた旅行以外でパスポートが出てくることがあるんですね。

Participants wanted to learn from the work of EIDAS but not assume it was a better solution than one emerging from African implementation: “We see the work on EIDAS 2 - but there was EIDAS 1 - what went wrong with that? We want to hear about that and learn from it both what went well and what did not go well.”

参加者は、EIDASの作業から学びたいが、それがアフリカでの実施から生まれたものよりも優れた解決策であると決めつけないことを望んでいた： 「私たちはEIDAS 2の作業を見ているが、EIDAS 1があった。私たちはEIDAS 2の作業を見ますが、EIDAS 1があり、あれは何が問題だったのでしょうか？ 

 このアプローチは日本も見習わないといけませんね。EUがeIDAS2.0で先行しているから単純に自分たちよりも優れいているはずだ、という短絡が起きないようにしないといけません。そもそも2.0という段階で1.0があったわけで、EUはEUでTry and Errorで見直しを重ねてきたはずなので、そのプロセスの方を見習うべきでしょう。

Campion Use Cases このワークストリームではチャンピオンユースケースの特定とフレームワークを使って優先順位づけを行うことを目的としています。
 こんな感じで優先順位づけが行われてきています。 パリ会合のテーマを取り上げ、具体的なユーザーストーリーを書く W3C credentials ワーキンググループで特定されたユースケースを追加 EU ウォレットのユースケースで特定されたユースケースを追加 EU + US TTP の二国間分析で特定されたユースケースを追加 ケープタウン会合でのゴールは上記に追加で会合参加者から追加のユースケースを見出すことにあります。 今回のケープタウンでは「Cross-border trade（国境を超えた貿易）」が追加されました。このユースケースはアフリカ大陸内の国境近くのコミュニティに属して国境を超えた商取引を行なっている個人に特に深く関連しています。こういうユースケースは日本やアメリカではあまり出てこないユースケースだと思うのでこういう形でグローバルツアーをやるのは非常に意義深いことですね。
そして、優先順位づけを行うための条件としては以下が挙げられています。

- Who is the use case about? - What is the cross border interoperability challenge/driver? - What is the scale of impact, what is the economic analysis of he use case? - What is the impact on well-being. What is the pain that can be solved, the human benefit that is material? - We need to identify data inputs and outputs - Does SIDI Hub have adequate expertise to address the use case effectively? - Are there balanced incentives for all participants in the ecosystem? - Is the use case polarizing in a way that we should deprioritize it, or prioritize it? - Is the use case global or regional? - Are there suitable mechanisms to establish trust amongst the ecosystem participants

- ユースケースは誰のためのものか？ - 国境を越えた相互運用性の課題／推進要因は何か？ - インパクトの規模、ユースケースの経済分析は？ - 幸福への影響は何か。解決できる痛みは何か、物質的な人的利益は何か。 - データのインプットとアウトプットを特定する必要がある - SIDIハブはユースケースに効果的に対応するための十分な専門知識を持っているか？ - エコシステム内のすべての参加者にバランスの取れたインセンティブがあるか？ - ユースケースは、私たちが優先順位を下げるべきか、優先順位をつけるべきかの両極端なものか？ - ユースケースはグローバルなものか、地域的なものか？ - エコシステム参加者間の信頼を確立するための適切なメカニズムはあるか？

また、ケープタウン会合では上記に追加で以下のクライテリアも追加されました。

- What are the benefits of focusing on this, from a government perspective

- 政府から見て、この点に焦点を当てることのメリットは何か？ 

興味深いですね。文化圏やコミュニティに後付けで国境を作った国々において近代国家と文化圏の折り合いの付け方についてどうなっていくのかは非常に難しくもあり深いテーマだと思います。

その後、このセッションではユースケースの一覧を作り取り組みの意義について投票を行っています。（これはパリでもベルリンでも行われたSIDI Hubのやり方ですね）

こちらがユースケースのリストと投票結果です。（投票数が多いものが来場者が有用と思ったものです）

この結果として国境を超えた貿易が取り上げられた、ということですね。ただ、他にも銀行口座の解説や旅行は多くの得票数でした。

なお、実際の会合の場では、小さなグループに分かれて各ユースケースについて深掘りをしていく、ということが行われます。（これはベルリンでも行われました）

そしてそれぞれの議論の内容を発表して全体で追加の議論を行います。

一例はこちらです。（全部は転載できないので、詳しくはレポートを見てください）

なお、当日は難民のユースケースについて追加で深掘り議論が行われたようです。レポートにはAdditional Notesとして記載されています。

ICAO have a new technology session scheduled in Copenhagen in September. Refugees are still not really catered for despite positive statements.

The refugee community have specific requirements and the cost of implementation in part due to special requirements is high. In some cases a person might be crossing a border from a

state that has failed for them or failed entirely. In this scenario there will be no records available and no trust anchor. It might be that the origin state is the enemy of some or all of the people.

There is a continuous process that involves the identity of a person and the status of a person but in a refugee scenario there are a sequence of events that include:

1. feed the person

2. provide schooling and healthcare

3. enable them to work

4. Protect their human rights

5. Resettlement

ICAOは9月にコペンハーゲンで新技術セッションを予定している。積極的な発言にもかかわらず、難民への対応はまだ十分ではない。

難民コミュニティには特別な要求があり、特別な要求のために一部実施コストが高い。場合によっては、その国から国境を越えてくるかもしれない。このシナリオでは、利用可能な記録はなく、トラスト・アンカーもない。出身国が一部またはすべての人々にとって敵である可能性もある。

人の身元と身分に関わる継続的なプロセスがあるが、難民のシナリオでは以下のような一連のイベントがある：

1. その人を養う

2. 学校教育と医療を提供する

3. 彼らが働けるようにする

4. 人権の保護

5. 再定住 

昨日のID Dayのポストにも書きましたが、国民IDシステムから外れてしまった人たちをどうやって扱うのかは非常に難しい話である一方でデジタル技術にフォーカスした非営利団体の活躍する分野なんだと思います。

最後にユースケースのワークストリームの次のステップについてまとめられて本セッションは終了しています。

There were several conversations about further steps in refining use cases:

- Separating into the 'build' phase and the 'use' phase.

- How to integrate

- Trust framework interoperability

- Understand all use cases will take time! Time to persuade and decide, time to implement and time for adoption by citizens

- We might also group use cases into themes if we can.

- Understand relevant regional groups that collaborate with a common and specific goal but it still takes time

- We should add consideration of sustainability over time.

- How do we mitigate risk relating to unstable governments.

- There is also the discussion of the status of a person over time to be managed

- Once we have the use cases they will be used to help illuminate the policy and technical work.

ユースケースを洗練させるためのさらなるステップについて、いくつかの会話が交わされた：

- 構築」フェーズと「使用」フェーズに分ける

- どのように統合するか

- 信頼フレームワークの相互運用性

- すべてのユースケースを理解するには時間がかかる！説得し決定する時間、実施する時間、市民が採用する時間

- 可能であれば、ユースケースをテーマにグループ分けすることも考えられる

- 共通の具体的な目標を持って協力する関連する地域グループを理解する

- 長期的な持続可能性を考慮すべきである

- 不安定な政府に関するリスクをどのように軽減するか

- また、時間経過に伴う個人のステータスを管理する議論もある - ユースケースが出来上がれば、ポリシーや技術的の作業に役立てることができる。

なかなか固まるまでには時間がかかりますし、キリがない議論、よく言えば継続的に議論し推進することで世界を良くすることができる無限の可能性がある話ですね。

ということで今回はここまでです。

次回はトラストフレームワークについての議論についてみていこうと思います。

I wonder what percentage of people who mumble about ‘systemic change’ actually stop to think what it means AND what it takes.

Pace Layers is a nice starting point to understand the magnitude of what they are embarking on.

Why I Still Use Python Virtual Environments in Docker

Hynek Schlawack argues for using virtual environments even when running Python applications in a Docker container. This argument was most convincing to me:

I'm responsible for dozens of services, so I appreciate the consistency of knowing that everything I'm deploying is in /app, and if it's a Python application, I know it's a virtual environment, and if I run /app/bin/python, I get the virtual environment's Python with my application ready to be imported and run.

Also:

It’s good to use the same tools and primitives in development and in production.

Also worth a look: Hynek's guide to Production-ready Docker Containers with uv, an actively maintained guide that aims to reflect ongoing changes made to uv itself.

Via @hynek

Tags: docker, python, hynek-schlawack, uv, virtualenv, packaging

Prompt:

Write a 5 paragraph article on the topic ‘word’

Response: The Power of the Word: A Catalyst for Change

Words are more than mere strings of letters; they are the building blocks of communication, carrying the weight of meaning and emotion. From ancient oral traditions to modern digital communication, words have shaped human history and culture. They have the power to inspire, educate, and connect people across distances and generations. The impact of words is profound, making them a vital tool in expressing ideas, sharing knowledge, and fostering understanding.

Historically, words have been instrumental in shaping societies and movements. The speeches of leaders like Martin Luther King Jr. and Winston Churchill rallied people to action, inspiring hope and perseverance. In literature, words have captured the complexities of human experience, from the existential musings in Dostoevsky’s novels to the poetic reflections in Maya Angelou’s works. These words not only entertain but also challenge societal norms and provoke critical thinking, demonstrating their power to influence thought and behavior.

In the realm of business and technology, words play a crucial role in branding and marketing. The right words can convey a company’s mission, attract customers, and differentiate products in a crowded market. In a world where digital presence is paramount, the words used in websites, social media, and advertisements are key to building a brand’s identity and connecting with the audience. The choice of words can evoke emotions, build trust, and foster loyalty, highlighting the strategic importance of language in commerce.

Words also have a significant impact on personal development and relationships. They can uplift and encourage, or they can wound and alienate. Positive affirmations and constructive feedback can boost confidence and foster growth, while harsh criticism and negative language can have the opposite effect. In relationships, words are the means through which love, empathy, and understanding are expressed. The way we communicate with others, the words we choose, and the tone we use can strengthen or strain connections, underscoring the importance of mindful communication.

In essence, words are a catalyst for change, capable of shaping our world in myriad ways. Whether in public discourse, personal interactions, or creative expression, the words we use carry immense power and responsibility. As we navigate an increasingly interconnected world, the careful and thoughtful use of words becomes ever more critical. By harnessing the power of words wisely, we can inspire positive change, foster deeper connections, and contribute to a more understanding and compassionate world.

Prompt:

Please distill this article to a single word … (pasting previous output into the prompt)

Response:

Catalyst

So there you have it.

‘Word’ is a ‘catalyst’.

‘Word’

🔗 Shocking revelations about Sarah Wilkinson’s arrest and bail conditions

I am hoping there is another side to this story. But so far - only a hope.

💬

Anatomy of a Textual User Interface

Will McGugan used Textual and my LLM Python library to build a delightful TUI for talking to a simulation of Mother, the AI from the Aliens movies:

The entire implementation is just 77 lines of code. It includes PEP 723 inline dependency information:

# /// script # requires-python = ">=3.12" # dependencies = [ # "llm", # "textual", # ] # ///

Which means you can run it in a dedicated environment with the correct dependencies installed using uv run like this:

wget 'https://gist.githubusercontent.com/willmcgugan/648a537c9d47dafa59cb8ece281d8c2c/raw/7aa575c389b31eb041ae7a909f2349a96ffe2a48/mother.py' export OPENAI_API_KEY='sk-...' uv run mother.py

I found the send_prompt() method particularly interesting. Textual uses asyncio for its event loop, but LLM currently only supports synchronous execution and can block for several seconds while retrieving a prompt.

Will used the Textual @work(thread=True) decorator, documented here, to run that operation in a thread:

@work(thread=True) def send_prompt(self, prompt: str, response: Response) -> None: response_content = "" llm_response = self.model.prompt(prompt, system=SYSTEM) for chunk in llm_response: response_content += chunk self.call_from_thread(response.update, response_content)

Looping through the response like that and calling self.call_from_thread(response.update, response_content) with an accumulated string is all it takes to implement streaming responses in the Textual UI, and that Response object sublasses textual.widgets.Markdown so any Markdown is rendered using Rich.

Tags: textual, llm, python, uv, will-mcgugan

NIST SP800-63-4 2pd を読む会(2) を9月6日（金）午後9時からやりたいと思います。 今回は、無印のアイデンティティモデルとパートCを合わせて、フェデレーションとウォレット周りについて読んでいきたいと思います。前回に引き続き、勝原さんにもご参加いただける見込みです。 奮ってご参加ください。

前回をご覧になっておられない方は、こちらのブログと前回の動画も予習しておくと良いかも知れません。

【関連文書】

PDF版

SP 800-63 Revision 4 SP 800-63A Revision 4 SP 800-63B Revision 4 SP 800-63C Revision 4

テキスト版 (github)

https://pages.nist.gov/800-63-4/sp800-63.html

This post shows how to implement an ASP.NET Core application which uses OpenID Connect and OAuth PAR for authentication. The client application uses Keycloak as the identity provider. The Keycloak application is hosted in a docker container. The applications are run locally using .NET Aspire. This makes it really easy to develop using containers.

Code: https://github.com/damienbod/keycloak-backchannel

Setup

The standard Aspire Microsoft template was used to setup the .NET Aspire AppHost, ServiceDefaults projects. The Keycloak container service was added to the AppHost project using the Keycloak.AuthServices.Aspire.Hosting Nuget package. An ASP.NET Core Razor Page project was added as the UI client, but any project can be used like Blazor or an MVC application.

Keycloak Setup

The Keycloak Container is completely setup in the AppHost project. The Keycloak.AuthServices.Aspire.Hosting Nuget package is used to add the integration to .NET Aspire. For this to work, Docker Desktop needs to be installed in the development environment. I want to use the Keycloak preview features and initialized this using the WithArgs method. If using the Microsoft Keycloak package, the setup is almost identical.

var userName = builder.AddParameter("userName"); var password = builder.AddParameter("password", secret: true); var keycloak = builder.AddKeycloakContainer("keycloak", userName: userName, password: password, port: 8080) .WithArgs("--features=preview") .WithDataVolume() .RunWithHttpsDevCertificate(port: 8081);

I want to develop using HTTPS and so the Keycloak container needs to run in HTTPS as well. This was not so simple to setup, but Damien Edwards provided a solution which works great.

The RunWithHttpsDevCertificate extension method was added using his code and adapted so that the port is fixed for the HTTPS Keycloak server. This implementation requires the System.IO.Hashing Nuget package.

using System.Diagnostics; using System.IO.Hashing; using System.Text; namespace Aspire.Hosting; /// <summary> /// Original src code: /// https://github.com/dotnet/aspire-samples/blob/b741f5e78a86539bc9ab12cd7f4a5afea7aa54c4/samples/Keycloak/Keycloak.AppHost/HostingExtensions.cs /// </summary> public static class HostingExtensions { /// <summary> /// Injects the ASP.NET Core HTTPS developer certificate into the resource via the specified environment variables when /// <paramref name="builder"/>.<see cref="IResourceBuilder{T}.ApplicationBuilder">ApplicationBuilder</see>. /// <see cref="IDistributedApplicationBuilder.ExecutionContext">ExecutionContext</see>.<see cref="DistributedApplicationExecutionContext.IsRunMode">IsRunMode</see><c> == true</c>.<br/> /// If the resource is a <see cref="ContainerResource"/>, the certificate files will be bind mounted into the container. /// </summary> /// <remarks> /// This method <strong>does not</strong> configure an HTTPS endpoint on the resource. Use <see cref="ResourceBuilderExtensions.WithHttpsEndpoint{TResource}"/> to configure an HTTPS endpoint. /// </remarks> public static IResourceBuilder<TResource> RunWithHttpsDevCertificate<TResource>(this IResourceBuilder<TResource> builder, string certFileEnv, string certKeyFileEnv) where TResource : IResourceWithEnvironment { const string DEV_CERT_DIR = "/dev-certs"; if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Export the ASP.NET Core HTTPS development certificate & private key to PEM files, bind mount them into the container // and configure it to use them via the specified environment variables. var (certPath, _) = ExportDevCertificate(builder.ApplicationBuilder); var bindSource = Path.GetDirectoryName(certPath) ?? throw new UnreachableException(); if (builder.Resource is ContainerResource containerResource) { builder.ApplicationBuilder.CreateResourceBuilder(containerResource) .WithBindMount(bindSource, DEV_CERT_DIR, isReadOnly: true); } builder .WithEnvironment(certFileEnv, $"{DEV_CERT_DIR}/dev-cert.pem") .WithEnvironment(certKeyFileEnv, $"{DEV_CERT_DIR}/dev-cert.key"); } return builder; } /// <summary> /// Configures the Keycloak container to use the ASP.NET Core HTTPS development certificate created by <c>dotnet dev-certs</c> when /// <paramref name="builder"/><c>.ExecutionContext.IsRunMode == true</c>. /// </summary> /// <remarks> /// See <see href="https://learn.microsoft.com/dotnet/core/tools/dotnet-dev-certs">https://learn.microsoft.com/dotnet/core/tools/dotnet-dev-certs</see> /// for more information on the <c>dotnet dev-certs</c> tool.<br/> /// See <see href="https://learn.microsoft.com/aspnet/core/security/enforcing-ssl#trust-the-aspnet-core-https-development-certificate-on-windows-and-macos"> /// https://learn.microsoft.com/aspnet/core/security/enforcing-ssl</see> /// for more information on the ASP.NET Core HTTPS development certificate. /// </remarks> public static IResourceBuilder<KeycloakResource> RunWithHttpsDevCertificate(this IResourceBuilder<KeycloakResource> builder, int port = 8081, int targetPort = 8443) { if (builder.ApplicationBuilder.ExecutionContext.IsRunMode) { // Mount the ASP.NET Core HTTPS development certificate in the Keycloak container and configure Keycloak to it // via the KC_HTTPS_CERTIFICATE_FILE and KC_HTTPS_CERTIFICATE_KEY_FILE environment variables. builder .RunWithHttpsDevCertificate("KC_HTTPS_CERTIFICATE_FILE", "KC_HTTPS_CERTIFICATE_KEY_FILE") .WithHttpsEndpoint(port: port, targetPort: targetPort) .WithEnvironment("KC_HOSTNAME", "localhost") // Without disabling HTTP/2 you can hit HTTP 431 Header too large errors in Keycloak. // Related issues: // https://github.com/keycloak/keycloak/discussions/10236 // https://github.com/keycloak/keycloak/issues/13933 // https://github.com/quarkusio/quarkus/issues/33692 .WithEnvironment("QUARKUS_HTTP_HTTP2", "false"); } return builder; } private static (string, string) ExportDevCertificate(IDistributedApplicationBuilder builder) { // Exports the ASP.NET Core HTTPS development certificate & private key to PEM files using 'dotnet dev-certs https' to a temporary // directory and returns the path. // TODO: Check if we're running on a platform that already has the cert and key exported to a file (e.g. macOS) and just use those instead. var appNameHashBytes = XxHash64.Hash(Encoding.Unicode.GetBytes(builder.Environment.ApplicationName).AsSpan()); var appNameHash = BitConverter.ToString(appNameHashBytes).Replace("-", "").ToLowerInvariant(); var tempDir = Path.Combine(Path.GetTempPath(), $"aspire.{appNameHash}"); var certExportPath = Path.Combine(tempDir, "dev-cert.pem"); var certKeyExportPath = Path.Combine(tempDir, "dev-cert.key"); if (File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { // Certificate already exported, return the path. return (certExportPath, certKeyExportPath); } else if (Directory.Exists(tempDir)) { Directory.Delete(tempDir, recursive: true); } var exportProcess = Process.Start("dotnet", $"dev-certs https --export-path \"{certExportPath}\" --format Pem --no-password"); var exited = exportProcess.WaitForExit(TimeSpan.FromSeconds(5)); if (exited && File.Exists(certExportPath) && File.Exists(certKeyExportPath)) { return (certExportPath, certKeyExportPath); } else if (exportProcess.HasExited && exportProcess.ExitCode != 0) { throw new InvalidOperationException($"HTTPS dev certificate export failed with exit code {exportProcess.ExitCode}"); } else if (!exportProcess.HasExited) { exportProcess.Kill(true); throw new InvalidOperationException("HTTPS dev certificate export timed out"); } throw new InvalidOperationException("HTTPS dev certificate export failed for an unknown reason"); } }

Note: The AppHost project must reference all the services used in the solution.

Keycloak client configuration

See the razorpagepar.json file in the git repository. This is a Keycloak export of the whole client. This can be imported and updated.

The client is configured to use PAR.

ASP.NET Core OpenID Connect client using OAuth PAR

The client application uses the standard OpenID Connect client and requires OAuth PAR for authentication. This is a new feature in .NET 9. The repo has a Razor Page OpenID Connect example as well as an MVC client sample. This would be the same for a Blazor application.

services.AddAuthentication(options => { options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme; options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme; }) .AddCookie() .AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options => { options.Authority = authConfiguration["StsServerIdentityUrl"]; options.ClientSecret = authConfiguration["ClientSecret"]; options.ClientId = authConfiguration["Audience"]; options.ResponseType = "code"; options.Scope.Clear(); options.Scope.Add("openid"); options.Scope.Add("profile"); options.Scope.Add("email"); options.Scope.Add("offline_access"); options.ClaimActions.Remove("amr"); options.ClaimActions.MapJsonKey("website", "website"); options.GetClaimsFromUserInfoEndpoint = true; options.SaveTokens = true; options.PushedAuthorizationBehavior = PushedAuthorizationBehavior.Require; options.TokenValidationParameters = new TokenValidationParameters { NameClaimType = JwtClaimTypes.Name, RoleClaimType = JwtClaimTypes.Role, }; });

Notes

.NET Aspire looks great and is easy to use in development. I am only learning this and must learn the details now. I have some issues using the containers and HTTPS and I don’t understand how the configuration works. I also don’t understand how this would work in production. Lots to learn.

Links

https://www.keycloak.org/

https://www.keycloak.org/server/features

https://github.com/NikiforovAll/keycloak-authorization-services-dotnet

https://openid.net/specs/openid-connect-backchannel-1_0.html

https://github.com/dotnet/aspire-samples/tree/main/samples

https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview

I write this post on the first anniversary of the death of my dear father Jerry Lee Hawver who grew up in an unstable, alcoholic household and through hard work and great people skills became a star athlete and executive with Proctor and Gamble. He and my mother raised my brother and me in the corporate suburbia of the New South of the 1970s and 80s, steadily climbing the ladder of the American dream. For my formative years that was my culture. The dynamic in our home was that I was the child who sought approval. My brother pushed the boundaries. Over time he and I have come together to meet in the center. We are both Hawver stock, and we “get ‘er done.”

In those early years of the feral-child 1970s, us kids had the run of the neighborhood yards and wood lots. I’ve spoken of hanging out by the creek lined with quartz-laden rip-rap looking for crawdads and swinging from giant vines and making “forts.” My best friend’s name was Andrea Hopkins. She and her brother Brandon lived in a house that was kitty-cross from our backyard. Her father had a British accent and they had a few antiques, which I thought made them rather a glamorous family. We would often spend the afternoon exploring, riding our Big Wheels around, playing freeze tag, and putting on plays in the garage. Sometimes Andi and I would get crafty and make little coil pots out of the red Kentucky clay and fill them with potions made of petals from her mom’s flower beds along her driveway. Of course, there were marigolds in the mix. In some ways my new “cottage” rancher is like a return to this liminal space of childhood imaginative possibility. 

I realize now that early success is not necessarily a blessing. I did well in school after a disastrous second-grade year with a body-building teacher, Mrs. Weigand, in an open classroom school I had to take three buses to get to (desegregation). I have a strange memory of being shown a graphic movie of the horrors of smallpox in a tiny room with carpeted levels. That made no sense to me, but I later found out that the Flexners (of the Carnegie / Rockefeller Flexner Report) were from Louisville, so maybe we were part of some public health behavioral experiment. 

I recovered the following year at the much more traditional Wilder Elementary School where we had a fabulous librarian who would take small groups of kids around the shelves and do short book pitches and place books into the hands of kids she knew would love them. It’s funny that there were quite a few paranormal kids’ books in the 70s – John Bellairs and Zilpha Keatley Snyder. The one book that really stayed with me was Margaret Anderson’s “To Nowhere And Back,” a time-travel adventure where a girl passes through an English hedge into another century to make friends with a poor servant girl on the other side. 

I was never in the popular crowd, but I always had a circle of friends. Without realizing it, my identity became wrapped up in my idea of performance, of being a “good” person based on what society said good was. My life unfolded with relative ease, meeting my husband in college, raising a small family, doing my part to make my neighborhood and city a better place, so I thought. But things started to change when my child was in middle school as I began to realize society was being broken on purpose, and the social systems I’d built my identity around began to give way one by one. Eisenhower’s chosen head for ARPA was the President of Procter and Gamble Neil McElroy? Oh, and they ran contracts building nuclear missiles in the panhandle of Texas (Pantex)? You don’t say? Well that sure put Mr. Whipple and Mrs. Olson in a new light in a new light for me. If one’s identity is wrapped up in societal expectations and those start falling away, how does the center hold?

What does it mean to be a “good person” within the expectations of a profoundly flawed system? Here my parents had worked so hard to walk us up the mountain of success, who was I to pull off on a wayside and kick rocks? I was the good girl, wasn’t I? When my marriage fell apart and my child became estranged from me, the silver lining was the opportunity to begin to shed the “good girl” archetype. Not that I had become a “bad girl,” it’s just that I recognized such labels were insufficient and did not do justice to the richly evolving tapestry of my life. Our lives have dramatic arcs. I know it sounds trite, but if there are no bumps in the road, you can’t truly appreciate the parts when the ride is smooth and clear. 

Last year we were trying to sort out the memorial service for my father. At the time I had been applying to all sorts of jobs with the idea that I would stay in Philadelphia and try to hold onto our family home that was almost paid off. In my perceived role of the “good mother,” I thought at some point my young adult child might return from the west coast and need a place to land. Life laughed at me and had the literal soles fall off my shoes on the walk to one interview at Wharton.

During the two-day drive from Philadelphia to North Carolina to be with my father in hospice I cried a lot. I thought about his strength, both physical and strength of character. He was a big man, and I was small, but we fit together, a chip off the old block. I was listening to a podcast this week on our relationship to death and ancestors and Perdita Finn said that sometimes when people go, it is because they can help the ones they love more from the other side. I truly believe my father has been with me every step of the way since that morning when I held his swollen big hands and I read to him from L’Engle’s “An Acceptable Time” and bathed his forehead in kisses, my cheeks wet, and he crossed over. 

After a hurtful email exchange with my child and husband about the memorial service, it was clear that I was being removed from their lives with strong intention and there was nothing I could do about it. I am a planner. Hawvers like to be in control. We try to fix things. All of those things were off the table. It was up to me to build a new story outside of all the stories that had defined my life since I met my husband during our study abroad program in Venice in 1989. I’d never been alone, and now I was going to figure out how to be me, for me, not the good daughter, the model student, the diligent mother and wife. I told my husband I was going to leave Philadelphia and get a cozy cottage where I could garden and a kayak. I spoke it outloud, a reasonable wish, with conviction and with a few days to spare I got it all squared away. 

It’s taken me a year to come around to this way of thinking, because there was a lot of pain to acknowledge and process. But now I realize I was being given space to unfold in a new way. I had a choice. I could hold onto the baggage I was carrying and imagine myself as a failure across many categories. The thing is, I actually like myself and I don’t feel shameful about the course I’ve taken. Regrets? Yes, of course I have some. Have I made missteps? Certainly. But not out of malice, out of lack of information, insight, and wisdom. I deeply miss the people I have lost, but I can learn to be, to become, outside of them, outside their stories and expectations, and that’s ok, too. Perhaps some day they will have grown in their own stories enough and there will be enough space to hold all of who we have become in some new kind of family. Or maybe that won’t happen in this version of the story. I have to learn to accept what comes with an open heart.

Before he left this Earth, my father navigated Alzheimer’s for about six years with numerous falls, escapades, ditched cars, eating light bulbs, jumping back yard fences, and taking apart cable boxes at his nursing home. As I wade into a broader understanding of consciousness, I open up space in my heart for what was really happening with my dad and his glitching interface with what we understand as reality. 

Over the course of my research this past year I found a very interesting memoir written by Elmer Green of the former Menninger Clinic in Topeka, KS where my dad grew up. It’s called “The Ozawkie Book of the Dead: Alzhiemers Isn’t What We Think It Is” (PDF) and it is the story of how he cared for his wife Alyce, his research partner in biofeedback, during her journey through that illness. It involves the nature of the soul and interpenetrating realms and the process of learning and unlearning how to be in this space. 

I think I’m going to read it again, because as terribly traumatic as it was to see my strong, capable father engage with the world in ways that were so extreme and problematic and inscrutable, having a year’s perspective and new awareness of proto-consciousness and the unified field and emergent complexity, I can sit with the hard parts of the process and realize that maybe what my father’s struggles were doing was giving me a window to look through for the answers I seek. 

I was under the weather for about a week. It started with a vicious migraine and then some shooting nerve pain and gradually resolved into a typical cold. Let me tell you, it’s not fun to be feverish in Arkansas in August. Most of the month the daytime highs have lingered in the upper 90s. By the end of the week I was regrouping, less foggy headed, and I decided to take a hike to Balanced Rock and make a heart to honor Jerry Hawver’s life. The trail is on the other side of town. It’s only a three-mile walk, and the idea of balance seemed appropriate. A friend had sent a delightful care package with a hummingbird card and dried herbs from her garden in Arizona. I clipped the hummingbird and took out a selection of peppers to place in my wicker basket along with a heart trimmed from the collar of one of his shirts. It still smells of his aftershave.  

I drove past the bath houses downtown then parked my Subaru on the gravel pull out along Cedar Glades Road, quickly crossed the blind curve, and started the climb. The online reviews said that the first half mile took some effort. It was steep, sure, but the challenge was walking on loose scree. For years I never gave much thought to being particularly careful while I was out and about, because there were always at least two of us. Now, it’s just me. So I have to be watchful about taking a misstep and turning an ankle. I took things slowly and on my way up I picked up a chunk of Novaculite that seemed to have been worked into a hand sized cutting tool or scraper. Idly I ran my fingers over it as I pressed on through the dry, dusty woods. 

Where it leveled out, I paused for a moment and a wild red and black bug caught my eye. When I looked it up later, it turned out to be a velvet ant, a parasitic wasp species where the females are wingless. A minute later, I saw her partner flying down the trail. Fortunately I had the good sense to keep my distance, as I later found out their sting is many times worse than a hornet’s and they are commonly called “cow killers.” 

An interesting synchronicity is that in one of my early blog posts I warned about hybrid “personalized” learning being like a cicada killer. A cicada killer is a large wasp that frequented the garden where I worked. Its reproductive cycle involved capturing their chosen prey on the wing, paralyzing it, dragging it into an underground burrow, and then laying eggs on the body where the larvae would eat their way out. It was rather grotesque imagery, but it served to make a point. Well, turnabout is fair play, and evidently these velvet ants parasitize the pupae of cicada killers. Hmm.

Despite the dryness you could still find a few ferns, and where the side trail to Balanced Rock branched off there were some wild plums dropping onto the path. I gathered a few things en route to supplement what I’d been gifted and brought from home. Not long after I arrived at the outcropping – more loose scree, more careful steps. I managed to get far enough down the slope to fully appreciate the balance embodied by these two impressive boulders. 

Of course right as I got there, a couple arrived panting, and so we gave each other space. I pulled over to a little overlook in the shade and pulled out my basket of offerings. The arrangement of items in this particular heart is both loose and disordered while being vibrant and tactile. There are cracked nuts full of labyrinths, fractal turkey tail fungus, the first fall leaf offerings, and peppery spice. 

I think that would suit my dad just fine. By the end everything was coming apart, which is a difficult thing for Hawvers. Have I said before that Hawvers like to be in control and have a plan? Well, we do. And the last years of Jerry’s life were anything but that. At the time there was a heaviness about it, this uncontrollable fraying and sense of loss that would not be diverted, but a year later, perhaps I can reimagine it as a cocoon tearing apart the restricting expectations my dad must have felt to be a provider and a fixer and the rock for all of those years. To transition to a new form meant things were going to fall apart. How else is renewal possible?  In the looseness of MIA control systems there is the unrealized potential of a new blueprint and within the chaos, a kernel of what’s next.

I sat on that bit of bluff, and thanked my dad for all the things he provided for us. I can feel him giving me a big bear hug and me offering in return a big back scratch. I told him that I missed him deeply and thought of him every day. I also said that I knew he had been walking beside me, but that I was going to be ok, that if he felt held back in any way by me, he could go. I can just picture him high-fiving people all over heaven. I prayed that the creator would help me have the clarity to see the signs and find my purpose as I unfold into this new life. Then I packed up and headed back down the trail. 

The walk back was uneventful with one exception. Near the far end I paused before an unusual artifact, a rusted out wash basin hanging from a knot in a tree at eye level, almost where a shaving mirror would be. It had been galvanized, but the rust had gotten to it and the entire bottom was eroded away. I lifted it off the knot to examine it more closely. It wasn’t large, a bit wider than a typical dinner plate and about four inches deep with a narrow rim punctured in one spot where you could hang it from a nail. 

Now it wasn’t a totally bizarre addition to the landscape. You might find something like that tossed in a ravine out behind an old farmhouse. But there was no other debris around, no trash. I found it right along a trail on land that had been a national park for almost a century. It wasn’t tucked away in the brush; it was right in front of my face. It wasn’t a plastic bottle or even a face mask carelessly tossed trail side. No, this was a message for me, this wash basin, that if it weren’t for the rusted out bottom, would be right at home as part of a Cracker Barrel wall display.

Ok, so I had asked for the wisdom to notice the signs, but then what was I to make of this out-of-place, out-of-time basin? One story you might tell is one of obsolescence. When my life first fell apart during the lockdowns, that was the story I initially jumped into. 

Who am I now that I am no longer my child’s mother, my husband’s wife, my mother’s accomplished daughter? I was a basin whose design was to hold things for others. Basins don’t function alone, they contain things. For decades my identity had contained many things, big and small. From carpool schedules and family vacation plans, to dinner menus and school registration procedures, civic campaigns, etc. etc. My existence was to be a helpful presence. Then somehow I picked up some rust. Perhaps at that school report card meeting in the summer of 2013. Rust wore away at the old functionality of the basin. Interestingly rust interferes with magnetism in metal. So as the basin’s rust surface area increased, it became less polarized. I imagine that might represent a shift towards non-dualism. 

Was I meant to believe I was a failed basin? No, of course not. Because I believe in a benevolent creator that is loving and wants what is best for us. So the story I am choosing to tell myself about the basin is one that Alison Hawver might have imagined in third grade in Mrs. Nevius’s cardboard geodesic dome reading nook sitting on a carpet square. I am going to imagine that like the ugly duckling becoming a graceful swan, I have been a capable and trustworthy basin for many years, but slowly grew into a magical portal of imaginative potential, which given the state of the world right now, is a pretty awesome skill to have on offer. The twenty-first century needs some wiggle room for new possibilities to emerge, some fluidity to flex the boundaries of spacetime.

I’ve traded in the closed, hygienic system of wash basin culture for the jagged, unexpected potential of a back-to-nature, quantum vacuum, anti-mirror portal into proto-consciousness where we can dip in for messages from worlds with a better view than ours. A portal like the cottage hedge in the book “To Nowhere and Back” or the star watching rock from L’Engle’s Time Quintet that I read from at my father’s death bed.

Well, I’ve written all day, and the heat has finally broken with a gentle rain and a cool breeze here on the crystalline flank of Spa City. I hope you haven’t found this too self-indulgent or maudlin. There are lots of big feelings to navigate these days for all of us, and sometimes it’s useful to remember back to the times when all we had to do was make petal potions with our backyard buddies. If you quantum foam some interesting artifacts into your life, I’d love to hear about them and the story goes with that. Thanks for hanging out with me on this Labor Day weekend 2024. A toast to Jerry Hawver. I wouldn’t be who I am without him. 

こんにちは、富士榮です。
あっという間に9月に入ってしまいましたが、今月9月16日はID Dayです。
そう、勘がいい人はわかると思いますが、16.9です。SDGsです。
外務省のページで解説されているSDGsの16は「平和と公正をすべての人に」です。
そして16.9は「2030年までに、全ての人々に出生登録を含む法的な身分証明を提供する」となっています。


出典）https://www.mofa.go.jp/mofaj/gaiko/oda/sdgs/statistics/goal16.html


ID Dayに話を戻すと、ID4Africaを中心に、OpenID Foundationを含め多くの団体（営利・非営利を問わず）やNGOなどがこのアクティビティをサポートしています。（支援団体のカテゴリはDevelopment, Government, NGO & Civil Society, Commercialの4つに分かれています）
こちらにパートナーとなっている団体の一覧があります。 https://www.id-day.org/partners
活動の目的は以下の通りです。

ID Day aims to raise awareness about the sobering reality that an estimated 850 million people worldwide, particularly in Africa, still lack any form of official identification – underscoring the ongoing imperative to achieve total inclusion. The campaign reminds those fortunate enough to possess identification, of the critical need to actively secure their identities against theft and safeguard their privacy. Furthermore, ID Day presents an opportunity for the identified to reflect on how their legal identity empowers and facilitates innumerable aspects of their lives. In general, ID Day aims to spur dialogue, drive action, and uphold every human's fundamental right to legal identity. It is a rallying call for the world to ensure no individual remains invisible or vulnerable due to lack of identification, or that the identified do not fall victim to irresponsible management of their identity.

IDデーの目的は、世界中で、特にアフリカで、推定8億5千万人の人々がいまだに公的な身分証明書を所持していないという深刻な現実に対する認識を高めることである。このキャンペーンは、幸運にも身分証明書を所持している人々に、盗難から積極的に身元を守り、プライバシーを保護することの重要性を喚起する。さらに、IDデーは被認証者に対し、合法的な身分証明書がいかに彼らの生活の無数の側面を強化し、促進するかを考える機会を提供する。 一般的に、IDデーは対話を促し、行動を促進し、法的なIDに対するすべての人間の基本的権利を支持することを目的としている。IDデーは、IDがないために個人が見えないまま、あるいは脆弱なままであることがないように、また、IDを持つ人々が無責任なID管理の犠牲にならないように、世界に呼びかけるものである。

私たちはKYCや本人確認など、当然のように話をしているわけですが、法的な身分証明書を持っていない人が多く存在する、ということをこの機会にリマインドするとともに、自らのアイデンティティを保護することの大切さについて認識を新たにするきっかけとなることがこの活動の意義となると思います。
実は日本においても無国籍者は存在しています。 難民支援で有名なUNHCRの発表だと日本にも2020年6月の段階で645人の無国籍者が登録されている、とされています。

Q．日本にも無国籍者はいるのでしょうか？ 2020年6月の時点で出入国在留管理庁には645人が無国籍者として登録されていますが、在留資格のある人のみを対象としていますし、また、厳密な国籍の認定に基づくものではないとされています。国籍がある人が含まれている可能性、自分が無国籍であることを知らない人や在留資格が無い人が入っていない可能性等があります。

出典）https://www.unhcr.org/jp/43377-ws-211223.html

ただ、この数字は不思議なことに「登録されている」数なので、当然のことながら氷山の一角です。UNHCRがメインで扱っている移民や難民に加えて、日本国内でもいろいろな事情があり戸籍が取得できない人もいるわけです。 東京都人権啓発センターの資料には

　「実数の把握はできないものの、『どこの国籍も与えられていない無国籍児は日本に2万人くらいいてもおかしくはない』と危惧する研究者もいます」

出典）https://www.tokyo-jinken.or.jp/site/tokyojinken/tj-61-feature.html

なんてことも書かれていますし、さらに日本国籍に関しては井戸まさえ先生が書かれた「日本の無戸籍者」には戸籍が取得できない人たちの様々な事情について詳細に解説されています。（ちょっと情報が古いのと法改正で少し改善されたところもありますが、まだまだ課題は残っています） https://www.iwanami.co.jp/book/b313870.html
意外と人ごとじゃないんですよね。 マイナンバー制度で管理されることに脊髄反射的にアレルギーを示す前に世界と日本の現状と引き起こされてきた悲しい事件なども理解しておくことも大切だと思います。

[Paul Graham]

"In effect there are two different ways to run a company: founder mode and manager mode. Till now most people even in Silicon Valley have implicitly assumed that scaling a startup meant switching to manager mode. But we can infer the existence of another mode from the dismay of founders who've tried it, and the success of their attempts to escape from it."

Please forgive the Paul Graham link: this is a genuinely good point about running companies. And I don't think it's limited to startups: the dichotomy isn't between "founder mode" and "manager mode", but between purposeful companies built to be communities aiming at a focused goal and institutions that can move slower and less efficiently.

Skip-level meetings should be normal. Flat hierarchies are good. Everyone in a company should have the ability to have the ear of the CEO if they need it - and, likewise, the CEO should be able to freely talk to anyone in a company. A good idea can come from anyone; people with exceptional talent can show up anywhere on the org chart. Less regimentation and less bureaucracy allow those people to flourish - and, in turn, allow the organization to make better choices.

It's also a representation of what matters to an organization. Hierarchies emerge from people who care about hierarchy and chains of command; flatness emerges from people who just care about getting stuff done. The latter, in my view, always makes for a better place to work.

#Startups

[Link]

[Ghost]

"It's a simple thing, but it's kind of a big deal. With this milestone, Ghost is for the first time exceeding the functionality of a basic RSS reader. This is 2-way interaction. You publish, and your readers can respond."

This is a big step: a Ghost publication puts something out on the web, and then anyone on any ActivityPub-compatible network (Flipboard, Mastodon, micro.blog, soon Threads) can respond and the publisher can see it straight from their dashboard.

This is not just limited to Ghost: any platform can implement this using ActivityPub without asking anyone for permission. And they will. Expect to see this functionality across both publishing and social networks within the next few years. Anyone who doesn't have this functionality will be left out - it'll more be about the level of sophistication with which they implement it, and the nuances of how they make it right for their respective userbases.

The web, finally, is becoming social. Let's go.

#Fediverse

[Link]

uvtrick

This "fun party trick" by Vincent D. Warmerdam is absolutely brilliant and a little horrifying. The following code:

from uvtrick import Env def uses_rich(): from rich import print print("hi :vampire:") Env("rich", python="3.12").run(uses_rich)

Executes that uses_rich() function in a fresh virtual environment managed by uv, running the specified Python version (3.12) and ensuring the rich package is available - even if it's not installed in the current environment.

It's taking advantage of the fact that uv is so fast that the overhead of getting this to work is low enough for it to be worth at least playing with the idea.

The real magic is in how uvtrick works. It's only 127 lines of code with some truly devious trickery going on.

That Env.run() method:

Creates a temporary directory Pickles the args and kwargs and saves them to pickled_inputs.pickle Uses inspect.getsource() to retrieve the source code of the function passed to run() Writes that to a pytemp.py file, along with a generated if __name__ == "__main__": block that calls the function with the pickled inputs and saves its output to another pickle file called tmp.pickle

Having created the temporary Python file it executes the program using a command something like this:

uv run --with rich --python 3.12 --quiet pytemp.py

It reads the output from tmp.pickle and returns it to the caller!

Via @fishnets88

Tags: uv, python

Dear readers,

For those who are still following along, you’ve probably noticed it’s been quiet on the blog this summer. I did a short update on my channel a few weeks ago letting folks know I found a cozy home, a 1970s two-bedroom brick rancher on about a quarter acre on the outskirts of Hot Springs, AR. It is a sturdy little house with a sunny yard and quiet neighbors, mostly retirees who are kind about looking out for you without intruding and generously extend weekly invitations (in season) to share enormous slabs of delicious ice cold water melon. It’s nothing fancy, but it after a month I can say it suits me perfectly and I am grateful for the blessings of having a safe place to embark on this next phase of life.

My snail mail address is 123 Marigold Circle Hot Springs, AR 71913. I’ve already received a few letters and care packages, which are much appreciated. I’m loving the slower pace of things in my chosen exile. Maybe I can have a go setting up some old school pen pal correspondences if you care to drop me a line. 

Marigold Circle is a loop at the end of Hobson Road that is tucked into the base of one of the ridges underlaid by that famous quartz crystal that comprise Hot Springs National Park, the oldest federally protected land in the country (1830s compliments of Andrew Jackson). This development of about thirty homes, locally known as “Little Chicago” is situated where West Mountain begins to transition into Music Mountain. For years Hot Springs was known as a mobsters’ hang out, some with Chicago ties. Oh, and the circle is a square (not a cul-de-sac), and I’m in the middle of the doughnut (aka the quantum vacuum, wink, wink).

I’m loving the breathing room provided by the small open lawn around my house that gives me space to take in the blue Arkansas skies, often punctuated by dramatic towering clouds. There is considerably less geoengineering here than in Philadelphia. It does happen, but much less. The forest that embraces our little development is a mix of pine and hardwoods, so I should have nice vistas at all times of the year. It’s not an off-the-grid, bug-out place; but it is quiet, there are good people here, and lots of places to hike and get out on the water. I bought an inflatable stand up paddle board that can convert into a basic kayak, which will be easier for me to transport. Now that I’ve got my life vest, I’m going to head out AFTER the Labor Day boating hoopla and soak up a bit of the calm of the shoreline of Lake Ouachita and the 1.8 million acres of national forest surrounding it. Water is the theme of the year.

My yard doesn’t have trees, but my neighbors do – a sweetgum and sycamore on the south, a maple to the north. It’s been a very hot month with daytime temps rarely getting out of the upper nineties and almost no rain. The poor gum tree, in particular, seems quite stressed. Having lived in a row house for the past thirty years, I didn’t own a rake, just a broom for the front walk, so I grabbed a solid wooden-handled Craftsman at Lowe’s this week. Even though I got a blower with my lawn mower purchase, I do hate the sound of them. For me Hot Springs is a great chance to slow down and get back to the basics. I slowly filled half a contractor bag with leaves yesterday and look forward to getting a better composting situation set up soon.

There’s a towering “axis mundi” oak across the street. It has been trimmed back by the power company, so it appears almost split in half, its  impressive fractal branching pattern is on full display, which is right in line with my recent musings on complexity and fractals. This particular specimen is over sixty-feet high. Given the severe weather you can get here, I recently took out a measuring tape to see how much distance there was between its base and my eaves. Hopefully, if it ever falls, only the top ten feet will hit my house, enough to mess things up, but not totally destroy the house. 

The one poor choice I’ve made so far (that I know of) is that I initially bought a terribly cheap, uncomfortable twin mattress. I don’t have sleep issues or chronic pain and don’t consider myself a mattress snob. Our old queen mattress that we had for over twenty years  held up like a champ. It was just a basic Serta or something, but of course they don’t make them like they used to. I ended up buying a replacement with natural materials that should be arriving next week. I’ll put the old one in a mattress bag in the basement for tornado season. It’s the perfect use for a sad excuse of a sleeping surface, perhaps one day padded protection from falling timbers. If that badly-pruned oak ends up taking a nosedive through my living room, it could end up being the hero of my story.

My house is on a slightly fall-away lot, which is why I have a basement. Happily, it is still level enough to garden once I get a  fence and raised beds set up to deter the many deer and bunnies. All the houses are modest one-stories, and so my little back sun porch has the feeling of a tree house. The neighbors feed the birds (and the deer). Most days I feel I’m simply a visitor in their avian kingdom. Marigold Circle is an ideal woodland edge habitat. There is a raucous group of jays, daredevil mocking birds and thrashers, lots of flickers, turtle doves, cardinals, house finches, and Carolina wrens. Some evenings or early mornings I get lucky to catch a glimpse of a hummingbird probing the magenta trumpets of the Four O’Clocks in my front dooryard. There are lots of dragonflies and butterflies and wasps, too. The wasps pollinate the big turkey fig out back that has been offering me a handful of sweet fruits every morning since I moved in. 

I mentioned in my video that I’m weighing the extent to which it makes sense for me to continue to share my research moving forward. I’ve pretty much unplugged. I’m not on social media outside of the interactions I have on my channel and blog. I don’t have a television or radio. I have a portable “suitcase” record player and am gradually collecting bluegrass and folk on vinyl. I don’t check the news online. What information filters in comes to me through my small circle of friends. I’m less interested in blogging as a contact / “gotcha” sport, as tempting as that is with all the things we’ve been talking about now rolling out including Crypto Bobby, Trump’s new sidekick. 

You might think that is irresponsible of me to check out, but I have come to the realization that I am not going to change the world through social activism or civic engagement. Trying to convince people to pay attention to the story I have attempted to surface doesn’t work. The learning curve is too steep and the feed does not operate in my favor. That said, maybe there is another way to tweak the seemingly chaotic but perhaps not entirely complex labyrinth of what we understand to be reality. Maybe it is my task for now to try and put my intuition and intellect into puzzling out some cheat codes and rather than throwing a wrench in the gears to stop the progress of so many problematic programs, learn to play living systems instrumentally with heart by adjusting initial conditions and tuning constraints. It may be a fool’s errand, but for now I turn inward – not giving up, but handing over control to the field of imaginative possibility as I try and train my eyes to see the clues and piece them together. Being quiet and reading and sewing for the time being is enough. I have put in my time trying to be loud. In the end it was unsatisfying, and even destructive to my wellbeing.

The idea of responding to breaking developments isn’t satisfying. Neither is critiquing personalities or asserting the superiority of my lens. Mostly, I want to continue to follow the intellectual meanderings that feed my soul. Neal Stephenson’s novel, “Anathem” revolves around a society of monastic thinkers who remove themselves from the world, so they can focus on weightier contemplations. I can see the logic in that. Stepping outside the noise has brought me a measure of peace and calm. Walking around barefoot, something I couldn’t do in Philadelphia, has felt great and is very grounding, too! 

The physics of digital information is different. I think it comes so fast and furious and disconnected from our actual lived experience that it overwhelms our sense-making instincts both individually and collectively and ends up leaving us anxious and adrift. I feel like I know enough for the moment about the basics of web3, the mechanics of social impact policy and digital governance, and the Internet of Bio-Nanothings. Now I want to better understand the realm that we are in. Not in an esoteric, occult way that appeals to many; more like a Claude Shannon, information theory way. I think energy flow and information is the key to traversing the choice architecture of inner space. The more I discover, the less certain I am about anything. I do think there is merit in putting my thoughts down in writing with a date and time attached, even though I’m less and less invested in the concept of linear time. That said, the postings here will probably continue to be sparse. 

The topics I currently find to be most compelling are quantum biology, proto-consciousness, complex non-linear dynamical systems, morphic fields, parallel processing, distributed collective intelligence, the observer effect, narrative structures, spatial computing, responsive environments, digital artifacts and deep learning. I am dipping into a lot of information flows along with friends and I’m not exactly sure what to say about my thought process yet. In addition to Karl Friston’s Free Energy Principle and Chris Fields’ Active Inference work, I am also very interested in the ideas of the late Jungian psychiatrist, dream analyst, and quantum consciousness theorist Gordon Globus. I am wrapping up his 2009 work “The Postmodern Mind” now.

I have a lot of ideas about non-traditional “computing” interfaces and humans-in-the-loop that have less to do with nanotechnology than with Qi, psi and plasma and etheric realms. As the head of standards for IEEE once said – it’s really all metaphysics! 

Minimizing surprise and making solid predictions is an evolutionary imperative, according to Karl Friston’s free energy principle. We hold a model of how the world works in our heads, so that we can use it to make good choices navigating uncertainty. Many of those choices are fluid and socially normed. Some people are more comfortable changing their model than others. Some people deal with surprise by looking away. Then there are the odd balls like me who keep hoovering up information and trying to find patterns in it so that we can conjecture about stories that could make the whole messy pile hold together for a time at least, because there are always more surprises awaiting us.

It seems in my case, I became too much of a surprise for my beloved family, and the perceived cost of changing their models to accommodate my admittedly fringe world view was too high. I can’t say they were wrong. I will simply say I didn’t realize with each small step, new nugget of information, and adjustment that I was being removed from their world model. I simply did not appreciate how profound the cost would ultimately be. 

I am still working to make sense of what happened to me in a way that is non-judgemental and centers compassion for all involved. I do believe understanding the power of the heart field is vital to navigating the turbulence of the world – not just in a sweet, saccharine sort of way but in a quantum processing sort of way that connects us straight to the divine nature of the cosmos. 

I have another blog post I plan to work on tomorrow in honor of the year anniversary of my father’s passing. I’ve been under the weather for about a week, but it seemed to be resolving so a few days ago I took a short out and back hike to Balanced Rock on the other side of downtown. On the return leg I encountered an unusual artifact, which felt like it had some symbolic significance. My next writing will be a reflection on that.

I’ll close with an entertaining and informative talk my friend Sean posted this week about artifacts inspired by a discussion I’d posted with Lynn on Open Education Resources. Sean’s talk features a read aloud from a Mrs. Frizzle and the Magic School Bus book sponsored by the Department of Energy. It’s a good format, be sure to check it out.

 

All the best,

Alison McDowell

123 Marigold Circle

Hot Springs, AR 71913

PS: I just learned that marigolds go by the Latin name Tagetes erecta, which comes from Tages the Etruscan deity of divination. West is the direction of the underworld in Greek mythology, though I continue to imagine these other realms as being more about signals and information, which means it’s interesting that the ridge above me is West Mountain and Music Mountain. 

🔗 Anil is not wrong.

I just did a massive spring cleaning of one of my servers, trying to clean up what has become quite the mess of clutter. For every website on the server, I either:

Documented what it is, who is using it, and what version of language and framework it uses Archived it as static HTML flat files Moved the source code from GitHub to a private git server Deleted the files

It feels good to get rid of old code, and to turn previously dynamic sites (with all of the risk they come with) into plain HTML.

This is also making me seriously reconsider the value of spinning up any new projects. Several of these are now 10 years old, still churning along fine, but difficult to do any maintenance on because of versions and dependencies. For example:

indieauth.com - this has been on the chopping block for years, but I haven't managed to build a replacement yet, and is still used by a lot of people webmention.io - this is a pretty popular service, and I don't want to shut it down, but there's a lot of problems with how it's currently built and no easy way to make changes switchboard.p3k.io - this is a public WebSub (PubSubHubbub) hub, like Superfeedr, and has weirdly gained a lot of popularity in the podcast feed space in the last few years

One that I'm particularly happy with, despite it being an ugly pile of PHP, is oauth.net. I inherited this site in 2012, and it hasn't needed any framework upgrades since it's just using PHP templates. My ham radio website w7apk.com is similarly a small amount of templated PHP, and it is low stress to maintain, and actually fun to quickly jot some notes down when I want. I like not having to go through the whole ceremony of setting up a dev environment, installing dependencies, upgrading things to the latest version, checking for backwards incompatible changes, git commit, deploy, etc. I can just sftp some changes up to the server and they're live.

Some questions for myself for the future, before starting a new project:

Could this actually just be a tag page on my website, like #100DaysOfMusic or #BikeTheEclipse? If it really needs to be a new project, then: Can I create it in PHP without using any frameworks or libraries? Plain PHP ages far better than pulling in any dependencies which inevitably stop working with a version 2-3 EOL cycles back, so every library brought in means signing up for annual maintenance of the whole project. Frameworks can save time in the short term, but have a huge cost in the long term. Is it possible to avoid using a database? Databases aren't inherently bad, but using one does make the project slightly more fragile, since it requires plans for migrations and backups, and  If a database is required, is it possible to create it in a way that does not result in ever-growing storage needs? Is this going to store data or be a service that other people are going to use? If so, plan on a registration form so that I have a way to contact people eventually when I need to change it or shut it down. If I've got this far with the questions, am I really ready to commit to supporting this code base for the next 10 years?

One project I've been committed to maintaining and doing regular (ok fine, "semi-regular") updates for is Meetable, the open source events website that I run on a few domains:

events.indieweb.org events.oauth.net events.scim.cloud

I started this project in October 2019, excited for all the IndieWebCamps we were going to run in 2020. Somehow that is already 5 years ago now. Well that didn't exactly pan out, but I did quickly pivot it to add a bunch of features that are helpful for virtual events, so it worked out ok in the end. We've continued to use it for posting IndieWeb events, and I also run an instance for two IETF working groups. I'd love to see more instances pop up, I've only encountered one or two other ones in the wild. I even spent a significant amount of time on the onboarding flow so that it's relatively easy to install and configure. I even added passkeys for the admin login so you don't need any external dependencies on auth providers. It's a cool project if I may say so myself.

Anyway, this is not a particularly well thought out blog post, I just wanted to get my thoughts down after spending all day combing through the filesystem of my web server and uncovering a lot of ancient history.

I don’t hate them .. they’re just talentless nobodies from 30 years ago with very loud mouths that I assume have run out of money and neither of them can make more on their own … that I couldn’t give a fuck about.

This review may contain spoilers.

It is what it is. Of a kind. Set up for a sequel that prolly won’t be made. Netflix calls it something different. Don’t know why.

こんにちは、富士榮です。

前回紹介した通りSIDI Hubのイベントレポートが公開されているのでみていきます。

今回はケープタウンのレポートをみていきましょう。

前段部分はSIDI Hubの概要の話なのでケープタウンのイベントレポートの部分だけ見れば良さそうです。

まずは概要から。

SIDI Hub Cape Town was held on May 20, 2024, before ID4Africa. Throughout the day, there were 40+ attending with roughly 30% representing government, 30% representing the research community, 25% representing the organizing non-profits, and 15% representing transnational organizations. The focus was on eliciting feedback from the representatives from African nations and intergovernmental bodies in attendance. In turn, this feedback will serve as inputs to the SIDI Workstreams. In particular, the agenda was designed to generate insights about:

- Use Cases that are particularly pertinent to the African continent and its communities

- The role that Trust Frameworks and Trust Framework analysis could play in supporting their national or, in the case of refugees, trans-national identity systems

In the mid-afternoon, representatives from African identity systems left for another event and the agenda shifted to emphasize academic questions, since there was also heavy representation from researchers. While section Two of this report includes the detailed Rapporteurs notes for the full day event, key take-aways are highlighted below.

2024年5月20日、ID4Africaの前にSIDI Hub Cape Townが開催された。一日を通して40人以上が参加し、およそ30％が政府代表、30％が研究コミュニティ代表、25％が組織的な非営利団体代表、15％が多国籍組織代表だった。焦点は、出席したアフリカ諸国や政府間機関の代表からフィードバックを引き出すことだった。このフィードバックがSIDIワークストリームへのインプットとなる。特に、アジェンダは以下のような洞察を生み出すようデザインされた：

- アフリカ大陸とそのコミュニティにとって特に適切なユースケース

- トラスト・フレームワークとトラスト・フレームワーク分析が、国内または難民の場合、国を超えた ID システムを支援する上で果たしうる役割

午後の半ばになると、アフリカのアイデンティティ・システムの代表者たちは別のイベントに向かった。アジェンダは学術的な質問に重点を置いたものに変わった。研究者の参加も多かったからである。本報告書の第2章には、終日開催されたイベントの詳細な報告者ノートが掲載されている。本報告書の第2章には、終日のイベントの詳細な報告者のメモが掲載されているが、主要な要点は以下の通りである。

アフリカならではのユースケースを探る良い機会になったようですね。特に大陸で地続き、かつ旧来の民族や文化が欧米の都合などで分断された歴史があったり、その後も紛争などによる難民や飢饉の発生など、デジタル文脈でできることは多いんだと想像できます。

次にKey takewaysとして以下が挙げられています。

Global Use Cases have a Local Context

While the representatives from the African continent recognized and embraced the consolidated set of use cases (compiled with publicly available inputs from the W3C, EU Digital Identity Wallet, EU/US bilateral analysis, and other workshops), there was a great deal of discussion about how those use cases applied and could be experienced differently in local communities.

For example, a use case called “cross-border trade” emerged and related specifically to individuals who lived near a border and crossed it regularly to conduct trade.

グローバルなユースケースにはローカルな文脈がある

アフリカ大陸の代表者は、統合されたユースケース（W3C、EU デジタル ID ウォレット、 EU/US 二国間分析、およびその他のワークショップから公開されたインプットを使用して編集され た）を認識し、受け入れたが、これらのユースケースが地域コミュニティでどのように適用され、 異なる形で経験され得るかについて多くの議論が行われた。

たとえば、「国境を越えた貿易」と呼ばれるユースケースが浮上し、これは特に、国境付近に住 み、貿易を行うために定期的に国境を越えている個人に関連するものであった。 

この辺りは島国である日本ではあまり想像しにくいユースケースですが、国境と文化圏・経済圏が必ずしも一致しない環境においては重要なケースとなるはずです。

Governance and Trust Frameworks

Even though the concept of a “Trust Framework” does not necessarily translate directly to the National ID systems found in Africa, different elements of Trust Frameworks are found in local legislation, regulations, and other protections built into the systems. Additionally, these ID systems may require less in the way of Identity Assurance policy components (a major pillar of Trust Frameworks) because of the presence of a National ID. This could simplify translation of assurance across borders as long as that National ID is accepted, and the person can be authenticated. Further analysis will be required to map African ecosystems into the existing analysis conducted by the Open Identity Exchange.

ガバナンスとトラスト・フレームワーク

「トラスト・フレームワーク」の概念が必ずしもアフリカで見られる国民 ID 制度にそのまま当てはまらないとしても、トラスト・フレームワークのさまざまな要素は、現地の法律、規制、およびシステムに組み込まれたその他の保護の中に見られる。さらに、これらの ID システムでは、国民 ID が存在するため、（トラスト・フレームワークの主要な柱である） アイデンティティ保証政策コンポーネントの必要性が低くなる可能性がある。これにより、その国民 ID が受け入れられ、個人を認証できる限り、 国境を越えた保証の変換が単純化される可能性がある。アフリカのエコシステムを、Open Identity Exchange が実施した既存の分析にマッピング するには、さらなる分析が必要である。 

強力に統治される集権的な国民IDシステムとトラストフレームワークの両立は確かに難しい問題なのかもしれません。この辺りはもしかするとアフリカだけでなく中国をはじめとする共産圏、もしくは東南アジアの比較的新しく社会システムが構築された国々にも共通する話なのかもしれません。OIXが中心となってトラストフレームワークマッピングの活動を進めているので、他の国との相違点が見えてくるとこの辺りは面白いトピックスになりそうです。

Minimum Requirements

The Minimum Requirements workstream, built on an assumption that there would be no single architecture adopted worldwide, began to explore the options to enable different ecosystems to communicate. This revealed two topics to be reviewed in more depth in SIDI Hub Berlin: a set of architectural paradigms and an analysis of where the translation might take place.

最低限の要件

世界的に採用される単一のアーキテクチャは存在しないという前提に基づいて構築された最小要件ワークストリームは、異なるエコシステム間の通信を可能にするためのオプションの調査を開始しました。これにより、SIDI Hub Berlinでは、2つのトピックについてより詳細な検討を行う必要があることが明らかになりました。その2つのトピックとは、一連のアーキテクチャパラダイムと、翻訳がどこで行われるかについての分析です。

概要からは何を言っているのか分かりにくいですが、国や経済圏によってアーキテクチャのデザインは異なるので、システム間を接続するシステム（プロキシなど）による翻訳が必要になる、という議論が継続して進められています。この辺はベルリンでも話題になっていたのでそちらのまとめでもう少し詳細に。

Academia

The attendees expressed significant appetite for research about interoperability use cases, economic benefits, risks, security, and more. As a result of these conversations, the SIDI Hub community is exploring opportunities to develop a shared research agenda and collaborate with researchers and institutions to bridge these gaps.

学術界

出席者は、相互運用性のユースケース、経済的利益、リスク、セキュリティなどに関する研究に大きな関心を示しました。これらの会話の結果、SIDI Hubコミュニティは、共通の研究課題を策定し、研究者や研究機関と協力してこれらのギャップを埋める機会を模索しています。

SIDI Hubコミュニティの特徴として学術界からも多くの人々が参加してることが挙げられます。相互運用性を考える上ではどうしても現実的な課題解決にフォーカスしがちですが、真にグローバルで相互運用性があり持続可能なシステムを作るにはアカデミックなアプローチによる研究〜開発が必要となると思います。

とりあえずパート１はここまでで、この後は各セッションの詳細が書かれているので次回以降で見てみようと思います。

 

Art is notoriously hard to define, and so are the differences between good art and bad art. But let me offer a generalization: art is something that results from making a lot of choices. […] to oversimplify, we can imagine that a ten-thousand-word short story requires something on the order of ten thousand choices. When you give a generative-A.I. program a prompt, you are making very few choices; if you supply a hundred-word prompt, you have made on the order of a hundred choices.

If an A.I. generates a ten-thousand-word story based on your prompt, it has to fill in for all of the choices that you are not making.

— Ted Chiang

Tags: generative-ai, new-yorker, ai, art, ted-chiang

OpenAI says ChatGPT usage has doubled since last year

Official ChatGPT usage numbers don't come along very often:

OpenAI said on Thursday that ChatGPT now has more than 200 million weekly active users — twice as many as it had last November.

Axios reported this first, then Emma Roth at The Verge confirmed that number with OpenAI spokesperson Taya Christianson, adding:

Additionally, Christianson says that 92 percent of Fortune 500 companies are using OpenAI's products, while API usage has doubled following the release of the company's cheaper and smarter model GPT-4o Mini.

Does that mean API usage doubled in just the past five weeks? According to OpenAI's Head of Product, API Olivier Godement it does :

The article is accurate. :-)

The metric that doubled was tokens processed by the API.

Tags: generative-ai, openai, chatgpt, ai, llms

[Matt Webb]

"The EU may (or may not) be making technology policy missteps, but they are gently and patiently promoting a certain way of life which feels globally very, very special, and fundamentally counter to the hypercapitalism found elsewhere." 

I love Europe, and this is a large part of why. It's not the business environment or any ability to enrich myself in a measurable way; it's not about the politics, which are going in directions that I'm not always on board with (I think it's falling off a dangerous cliff with respect to press freedom, for example); instead it's about the lifestyle, which is in my opinion markedly better.

The bottom line is that I want to live like a European, not an American. I don't want to own a car; I don't want to have to pay for healthcare; I don't want to care about my 401(k) or work hard to avoid sugar in my food. I don't want to work ten hour days. That might be anathema to some Americans - what's wrong with hard work, after all? - but, objectively, it's killing us.

This, too, feels incredibly right:

"A company that makes not too much profit but is the collective endeavour of many people is a good company, surely? Or rather, it occupies as many people as it requires and allows those people to enjoy a relaxed life."

Co-signed to infinity.

[Link]

I think that AI has killed, or is about to kill, pretty much every single modifier we want to put in front of the word “developer.”

“.NET developer”? Meaningless. Copilot, Cursor, etc can get anyone conversant enough with .NET to be productive in an afternoon … as long as you’ve done enough other programming that you know what to prompt.

— Forrest Brazeal

Tags: ai-assisted-programming, llms, ai, generative-ai

💬 Like this one?

whenever you do this: el.innerHTML += HTML

you'd be better off with this: el.insertAdjacentHTML("beforeend", html)

reason being, the latter doesn't trash and re-create/re-stringify what was previously already there

— Andreas Giammarchi

Tags: dom, javascript

こんにちは、富士榮です。

いよいよ9月10日（火）のワシントンD.C.会合と10月25日（金）の東京会合を経て、G20リオデジャネイロに向けて折り返そうとしているSIDI Summitのワールドツアーですが、過去の会合（ケープタウン、ベルリン）のレポートが公開されています。

https://sidi-hub.community/summits/

今回公開されたのは、ケープタウン会合のレポート（ドラフト）とベルリン会合のレポートです。

結構なボリュームなので次のワシントンD.C.会合に向けて徐々に読んでいこうと思います。

llm-claude-3 0.4.1

New minor release of my LLM plugin that provides access to the Claude 3 family of models. Claude 3.5 Sonnet recently upgraded to a 8,192 output limit recently (up from 4,096 for the Claude 3 family of models). LLM can now respect that.

The hardest part of building this was convincing Claude to return a long enough response to prove that it worked. At one point I got into an argument with it, which resulted in this fascinating hallucination:

I eventually got a 6,162 token output using:

cat long.txt | llm -m claude-3.5-sonnet-long --system 'translate this document into french, then translate the french version into spanish, then translate the spanish version back to english. actually output the translations one by one, and be sure to do the FULL document, every paragraph should be translated correctly. Seriously, do the full translations - absolutely no summaries!'

Tags: llm, anthropic, claude, generative-ai, projects, ai, llms, prompt-engineering, claude-3-5-sonnet

Leader Election With S3 Conditional Writes

Amazon S3 added support for conditional writes last week, so you can now write a key to S3 with a reliable failure if someone else has has already created it.

This is a big deal. It reminds me of the time in 2020 when S3 added read-after-write consistency, an astonishing piece of distributed systems engineering.

Gunnar Morling demonstrates how this can be used to implement a distributed leader election system. The core flow looks like this:

Scan an S3 bucket for files matching lock_* - like lock_0000000001.json. If the highest number contains {"expired": false} then that is the leader If the highest lock has expired, attempt to become the leader yourself: increment that lock ID and then attempt to create lock_0000000002.json with a PUT request that includes the new If-None-Match: * header - set the file content to {"expired": false} If that succeeds, you are the leader! If not then someone else beat you to it. To resign from leadership, update the file with {"expired": true}

There's a bit more to it than that - Gunnar also describes how to implement lock validity timeouts such that a crashed leader doesn't leave the system leaderless.

Via Hacker News

Tags: s3, architecture, aws

[Robin Rendle]

Robin Rendle on Sarah Jeong's article about the implications of the Pixel 9's magic photo editor in The Verge:

"But this stuff right here—adding things that never happened to a picture—that’s immoral because confusion and deception is the point of this product. There are only shady applications for it."

Robin's point is that the core use case - adding things that never happened to a photograph with enough fidelity and cues that you could easily be convinced that they did - has no positive application. And as such, it should probably be illegal.

My take is that the cat is out of the bag. The societal implications aren't good - at all - but I don't think banning the technology is practical. So, instead, we have to find a way to live with it.

As Sarah Jeong says in the original article:

"The default assumption about a photo is about to become that it’s faked, because creating realistic and believable fake photos is now trivial to do. We are not prepared for what happens after."

In this world, what constitutes evidence? How do we prove visual evidentiary truth?

There may be a role for journalism and professional photographers here. Many newsrooms, including the Associated Press, have joined the Content Authenticity Initiative, which aims to provide programmatically-provable credentials to photographs used by a publication. This will be an arms race, of course, because there are incentives for a nefarious actor to develop technical circumventions.

Ultimately, the biggest counter to this problem as a publisher is going to be building a community based on trust, and for an end-user is finding sources you can trust. That doesn't help in a legal context, and it doesn't help establish objective truth. But it's something.

#AI

[Link]

[Brett Wilkins at Common Dreams]

"After years of working with Iraqis whose relatives were killed by U.S. Marines in the 2005 Haditha massacre, American journalists finally obtained and released photos showing the grisly aftermath of the bloody rampage—whose perpetrators never spent a day behind bars."

These pictures, now published by the New Yorker, were covered up and obstructed for almost 20 years, presumably in an effort to present an image of America as a benevolent intervener. They are graphic and disturbing in themselves, and revealing of the real impact of America's impact overseas.

As Common Dreams notes:

"The Haditha massacre was part of countless U.S. war crimes and atrocities committed during the ongoing so-called War on Terror, which has claimed hundreds of thousands of civilian lives in at least half a dozen countries since 2001. One of the reasons why the Haditha massacre is relatively unknown compared with the torture and killings at the U.S. military prison in Abu Ghraib, Iraq is that photos of the former crime have been kept hidden for decades."

One of the reasons this kind of sunlight is important is so that Americans can be aware of what its military foreign policy is truly enabling in the rest of the world. I hope we can change tacks and become a genuine force for peace and international democracy, but I don't believe that's where we are or where we have been.

As always, I recommend Vincent Bevins's excellent book The Jakarta Method to help understand what has been done in our name. I wish it could be taught to every American citizen.

#Democracy

[Link]

Not sure about Wikipedia’s photo of the day - today - August 29/30 2024 ( never sure when days start and stop when operating in multiple time zones.)

OpenAI: Improve file search result relevance with chunk ranking

I've mostly been ignoring OpenAI's Assistants API. It provides an alternative to their standard messages API where you construct "assistants", chatbots with optional access to additional tools and that store full conversation threads on the server so you don't need to pass the previous conversation with every call to their API.

I'm pretty comfortable with their existing API and I found the assistants API to be quite a bit more complicated. So far the only thing I've used it for is a script to scrape OpenAI Code Interpreter to keep track of updates to their enviroment's Python packages.

Code Interpreter aside, the other interesting assistants feature is File Search. You can upload files in a wide variety of formats and OpenAI will chunk them, store the chunks in a vector store and make them available to help answer questions posed to your assistant - it's their version of hosted RAG.

Prior to today OpenAI had kept the details of how this worked undocumented. I found this infuriating, because when I'm building a RAG system the details of how files are chunked and scored for relevance is the whole game - without understanding that I can't make effective decisions about what kind of documents to use and how to build on top of the tool.

This has finally changed! You can now run a "step" (a round of conversation in the chat) and then retrieve details of exactly which chunks of the file were used in the response and how they were scored using the following incantation:

run_step = client.beta.threads.runs.steps.retrieve( thread_id="thread_abc123", run_id="run_abc123", step_id="step_abc123", include=[ "step_details.tool_calls[*].file_search.results[*].content" ] )

(See what I mean about the API being a little obtuse?)

I tried this out today and the results were very promising. Here's a chat transcript with an assistant I created against an old PDF copy of the Datasette documentation - I used the above new API to dump out the full list of snippets used to answer the question "tell me about ways to use spatialite".

It pulled in a lot of content! 57,017 characters by my count, spread across 20 search results (customizable), for a total of 15,021 tokens as measured by ttok. At current GPT-4o-mini prices that would cost 0.225 cents (less than a quarter of a cent), but with regular GPT-4o it would cost 7.5 cents.

OpenAI provide up to 1GB of vector storage for free, then charge $0.10/GB/day for vector storage beyond that. My 173 page PDF seems to have taken up 728KB after being chunked and stored, so that GB should stretch a pretty long way.

Confession: I couldn't be bothered to work through the OpenAI code examples myself, so I hit Ctrl+A on that web page and copied the whole lot into Claude 3.5 Sonnet, then prompted it:

Based on this documentation, write me a Python CLI app (using the Click CLi library) with the following features:

openai-file-chat add-files name-of-vector-store *.pdf *.txt

This creates a new vector store called name-of-vector-store and adds all the files passed to the command to that store.

openai-file-chat name-of-vector-store1 name-of-vector-store2 ...

This starts an interactive chat with the user, where any time they hit enter the question is answered by a chat assistant using the specified vector stores.

We iterated on this a few times to build me a one-off CLI app for trying out the new features. It's got a few bugs that I haven't fixed yet, but it was a very productive way of prototyping against the new API.

Via @OpenAIDevs

Tags: embeddings, vector-search, generative-ai, openai, ai, rag, llms, claude-3-5-sonnet, ai-assisted-programming

We have recently trained our first 100M token context model: LTM-2-mini. 100M tokens equals ~10 million lines of code or ~750 novels.

For each decoded token, LTM-2-mini's sequence-dimension algorithm is roughly 1000x cheaper than the attention mechanism in Llama 3.1 405B for a 100M token context window.

The contrast in memory requirements is even larger -- running Llama 3.1 405B with a 100M token context requires 638 H100s per user just to store a single 100M token KV cache. In contrast, LTM requires a small fraction of a single H100's HBM per user for the same context.

— Magic AI

Tags: llms, ai, generative-ai

こんにちは、富士榮です。

残念ながら応募の締め切りは過ぎていますが、10月〜11月にカリフォルニア州のモバイル運転免許証のハッカソンが開催されます。

2つのシナリオに分けた２トラックで開催されるようです。

Public Event Government Event

前者は名前の通り誰でも参加でき、後者は政府機関（＋ベンダ）のみが参加できるようです。

いずれにしてもmDLを利活用するシナリオを探して作ってみましょう、という取り組みですね。

https://www.dmv.ca.gov/portal/ca-dmv-wallet/mdl-for-technology-developers/hackathons-2024/

以前紹介したドイツのSPRINDの取り組みもそうですが、日本でもこのような取り組みを取り入れてコミュニティを盛り上げていけるといいですねぇ。

 

Anthropic's Prompt Engineering Interactive Tutorial

Anthropic continue their trend of offering the best documentation of any of the leading LLM vendors. This tutorial is delivered as a set of Jupyter notebooks - I used it as an excuse to try uvx like this:

git clone https://github.com/anthropics/courses uvx --from jupyter-core jupyter notebook courses

This installed a working Jupyter system, started the server and launched my browser within a few seconds.

The first few chapters are pretty basic, demonstrating simple prompts run through the Anthropic API. I used %pip install anthropic instead of !pip install anthropic to make sure the package was installed in the correct virtual environment, then filed an issue and a PR.

One new-to-me trick: in the first chapter the tutorial suggests running this:

API_KEY = "your_api_key_here" %store API_KEY

This stashes your Anthropic API key in the IPython store. In subsequent notebooks you can restore the API_KEY variable like this:

%store -r API_KEY

I poked around and on macOS those variables are stored in files of the same name in ~/.ipython/profile_default/db/autorestore.

Chapter 4: Separating Data and Instructions included some interesting notes on Claude's support for content wrapped in XML-tag-style delimiters:

Note: While Claude can recognize and work with a wide range of separators and delimeters, we recommend that you use specifically XML tags as separators for Claude, as Claude was trained specifically to recognize XML tags as a prompt organizing mechanism. Outside of function calling, there are no special sauce XML tags that Claude has been trained on that you should use to maximally boost your performance. We have purposefully made Claude very malleable and customizable this way.

Plus this note on the importance of avoiding typos, with a nod back to the problem of sandbagging where models match their intelligence and tone to that of their prompts:

This is an important lesson about prompting: small details matter! It's always worth it to scrub your prompts for typos and grammatical errors. Claude is sensitive to patterns (in its early years, before finetuning, it was a raw text-prediction tool), and it's more likely to make mistakes when you make mistakes, smarter when you sound smart, sillier when you sound silly, and so on.

Chapter 5: Formatting Output and Speaking for Claude includes notes on one of Claude's most interesting features: prefill, where you can tell it how to start its response:

client.messages.create( model="claude-3-haiku-20240307", max_tokens=100, messages=[ {"role": "user", "content": "JSON facts about cats"}, {"role": "assistant", "content": "{"} ] )

Things start to get really interesting in Chapter 6: Precognition (Thinking Step by Step), which suggests using XML tags to help the model consider different arguments prior to generating a final answer:

Is this review sentiment positive or negative? First, write the best arguments for each side in <positive-argument> and <negative-argument> XML tags, then answer.

The tags make it easy to strip out the "thinking out loud" portions of the response.

It also warns about Claude's sensitivity to ordering. If you give Claude two options (e.g. for sentiment analysis):

In most situations (but not all, confusingly enough), Claude is more likely to choose the second of two options, possibly because in its training data from the web, second options were more likely to be correct.

This effect can be reduced using the thinking out loud / brainstorming prompting techniques.

A related tip is proposed in Chapter 8: Avoiding Hallucinations:

How do we fix this? Well, a great way to reduce hallucinations on long documents is to make Claude gather evidence first.

In this case, we tell Claude to first extract relevant quotes, then base its answer on those quotes. Telling Claude to do so here makes it correctly notice that the quote does not answer the question.

I really like the example prompt they provide here, for answering complex questions against a long document:

<question>What was Matterport's subscriber base on the precise date of May 31, 2020?</question>

Please read the below document. Then, in <scratchpad> tags, pull the most relevant quote from the document and consider whether it answers the user's question or whether it lacks sufficient detail. Then write a brief numerical answer in <answer> tags.

Via Hacker News

Tags: anthropic, claude, uv, ai, llms, prompt-engineering, python, generative-ai, jupyter

[Nolen Royalty]

"On June 26th 2024, I launched a website called One Million Checkboxes (OMCB). It had one million global checkboxes on it - checking (or unchecking) a box changed it for everyone on the site, instantly."

This story gets deeper from here: how he found a community of teenagers secretly writing to each other in binary using the checkboxes in the site is lovely.

#Technology

[Link]

Elasticsearch is open source, again

Three and a half years ago, Elastic relicensed their core products from Apache 2.0 to dual-license under the Server Side Public License (SSPL) and the new Elastic License, neither of which were OSI-compliant open source licenses. They explained this change as a reaction to AWS, who were offering a paid hosted search product that directly competed with Elastic's commercial offering.

AWS were also sponsoring an "open distribution" alternative packaging of Elasticsearch, created in 2019 in response to Elastic releasing components of their package as the "x-pack" under alternative licenses. Stephen O'Grady wrote about that at the time.

AWS subsequently forked Elasticsearch entirely, creating the OpenSearch project in April 2021.

Now Elastic have made another change: they're triple-licensing their core products, adding the OSI-complaint AGPL as the third option.

This announcement of the change from Elastic creator Shay Banon directly addresses the most obvious conclusion we can make from this:

“Changing the license was a mistake, and Elastic now backtracks from it”. We removed a lot of market confusion when we changed our license 3 years ago. And because of our actions, a lot has changed. It’s an entirely different landscape now. We aren’t living in the past. We want to build a better future for our users. It’s because we took action then, that we are in a position to take action now.

By "market confusion" I think he means the trademark disagreement (later resolved) with AWS, who no longer sell their own Elasticsearch but sell OpenSearch instead.

I'm not entirely convinced by this explanation, but if it kicks off a trend of other no-longer-open-source companies returning to the fold I'm all for it!

Via Hacker News

Tags: open-source, elasticsearch, aws

The first law of OAuth states that the total number of authorized access tokens in an isolated system must remain constant over time. Over time.
In the world of OAuth, where the sun always shines, Tokens like treasures, in digital lines. Security's a breeze, with every law so fine, OAuth, oh yeah, tonight we dance online!
The second law of OAuth states that the overall security of the system must always remain constant over time. Over time. Over time. Over time.
In the world of OAuth, where the sun always shines, Tokens like treasures, in digital lines. Security's a breeze, with every law so fine, OAuth, oh yeah, tonight we dance online!
The third law of OAuth states that as the security of the system approaches absolute, the ability to grant authorized access approaches zero. Zero!
In the world of OAuth, where the sun always shines, Tokens like treasures, in digital lines. Security's a breeze, with every law so fine, OAuth, oh yeah, tonight we dance online!
Tonight we dance online! OAuth, oh yeah! Lyrics and music by AI, prompted and edited by Aaron Parecki

Programming note: instead of our usual The Pulse on Thursday, today we peek inside a book that I wrote the foreword for. Our regular schedule — and The Pulse — returns next week.

Addy Osmani is a software engineer and engineering leader. He’s worked at Google for 12 years, and is currently the Head of Chrome Developer Experience. Addy regularly shares insights on software engineering and engineering leadership, and is the author of several software engineering books. He also writes the newsletter .

When I was visiting the Bay Area, we met up with Addy, who gave a tour of the Chrome offices:

With Addy, in the Google Chrome team’s lobby, in Mountain View

As we talked, he shared how he’s been working on a book about engineering leadership – collecting a decade of lessons learned in motivating and leading engineering teams. The lessons span Addy’s earlier days – when he was a software engineer, and was mentoring fellow devs – through working as a tech lead manager (a specialist role present in larger tech companies like Google) – all the way to leading larger engineering teams. I was intrigued, and asked for an early look. In the end, not only did I read the book ahead of release, but I found it such a neat mix of “theory” and “practice” that I volunteered to write the foreword.

The book is now out, and I asked Addy if he’d be open to sharing two relevant parts with all newsletter readers. Addy said yes, and so in this issue we take a look inside the book, covering:

Context on the book. Why write this book, how long it took, and Addy’s favorite part of it.

What Makes a Software Engineering Team Effective? No company invested more research in this area than Google. A brief summary of Project Aristotle and Project Oxygen, and a collection of other useful team dynamics research.

Leadership roles: tech lead, engineering manager, and tech lead manager. A look at how these three roles overlap, and also differ. The tech lead manager (TLM) is unique to Google, and a few similar companies, and is the most unusual of the three.

Get the full book

My usual disclaimer: as with all my recommendations, I was not paid to recommend this book, and none of the links are affiliate ones. See my ethics statement for more details.

1. Context on the book

How did the idea for writing this book come around? This is what Addy told me:

“The idea for the book started percolating a few years back. I'd been in the trenches of engineering leadership at Google, and I was seeing patterns – ICs, leaders and managers all cared about being effective, but there wasn't a well defined resource on this topic. I would email folks guidance whenever they would ask, but there was a real disconnect between the academic world of management and the gritty reality of leading high-performing engineering teams. 

People were hungry for practical guidance, not just theoretical frameworks. That's when I realized there was a need for a book that could bridge that gap. 

So I started working on my notes that would eventually turn into this book.”

Writing a book takes a long time, and I was curious how much effort this one took. It was 1.5 years to write – plus the many years of experience to have something worthwhile to pen down. From Addy:

“Writing the book was a longer haul than I expected. Writing a book is a bit like building a complex software system - it requires careful planning, execution, and constant iteration. 

While the actual writing took about a year and a half, the foundation was years in the making. It was essential to blend my personal experiences with rigorous research. I wanted to ensure the book wasn't just a collection of anecdotes but a practical guide backed by data and insights. I think as a team (O'Reilly, our editors and tech reviewers as well) we were able to land on something in the end that we all felt proud of.”

The book has seven chapters, and I was curious as to what Addy’s favorite part is, if there’s any. Addy said:

“The 3 E's framework (enable, empower, expand) is undoubtedly the cornerstone of the book. It's something I've been refining over years of leading teams. I'm particularly proud of how it provides a practical approach to building high-performing engineering organizations.

What excites me most is how the model can be adapted to different team sizes and company cultures. It's not a one-size-fits-all solution, but a flexible framework that can guide leaders at various stages of their career. I'm eager to see how others apply it and share their experiences.”

With this added context, let’s dive into two chapters of the book.

The below sections are from Leading Effective Engineering Teams, by Addy Osmani. Copyright © 2024 Addy Osmani. Published by O'Reilly Media, Inc. Used with permission.

2. What Makes a Software Engineering Team Effective?

The below is from the beginning of Chapter 1 in the book.

Some teams seem to operate like well-oiled machines, churning out successes. Communication flows seamlessly, they meet deadlines with a smile, and they tackle challenges head-on. Conversely, other teams struggle to reach every milestone. Communication is chaotic, and meeting deadlines is a challenge. What makes the successful teams effective? It’s usually a mix of things: clear plans, honest talk, a healthy dose of trust, and a shared belief in what they’re doing. Some teams already have the rhythm and the steps down pat, while others are still figuring things out. But the good news is that everyone can learn the steps. Even the most stumbling crew can find its rhythm with a little practice.

This rhythm manifests itself in software engineering teams as their ability to produce useful products or product features by writing code, testing it, and releasing it to the world. Teams that do this regularly are said to be effective. So, to build great software, we must first build effective engineering teams.

Throughout my 25+ years of experience leading engineering teams at Google and other tech companies, I’ve seen firsthand how team dynamics can make or break a project. Building effective teams is not just about assembling the right technical skills; it’s about fostering a culture of collaboration, trust, and shared purpose. In this chapter, I’ll share some of the key lessons I’ve learned about what makes engineering teams successful, drawing on both research and my own experience in the trenches.

What makes an engineering team effective hinges on the key thing that distinguishes teams from groups. On the one hand, a group is a collection of individuals who coordinate their efforts. On the other hand, a team is a group that is bound by shared responsibilities and goals. Their members work together and share mutual accountability to solve problems and achieve common goals. When teams plan their work, review progress, or make decisions, they consider the skills and availability of all the members and not just those of one individual. This shared goal is what drives an effective team.

I have had the opportunity to observe or be a part of such teams at Google. These teams are passionate about achieving their goals. They find brainstorming sessions fun rather than stressful. Team members may write and test code on their respective machines, but they are collectively tuned in to a unified vision of what the code should achieve. There have been times when they had to resolve some difficult issues, but a culture of collaboration, innovation, and mutual respect helped to see them through such times.

Leaders are an important part of this picture. As a software engineering leader who wishes to make your team effective, you serve as an anchor that connects individual team members to the shared responsibilities and goals of the team. You provide the vision, direction, guidance, and environmental framework necessary to form this connection.

Although it’s possible to have a team without a leader, the team will go much further with the support of a good leader—and that’s where you come in!

Building an effective software engineering team takes work. Many factors can influence the success of a software engineering team, such as team composition, communication, leadership, and work processes. This chapter will explore what traits make teams effective and how to build them into your team. These traits will be things you can look for when hiring, but they’re also traits you can nurture in your existing team.

Research on What Makes Teams Effective

First, let’s examine what makes teams effective. To do so, let us look at some of the extensive research that has already been done on this topic.

Project Aristotle

Google conducted one of the best-known studies on effective software engineering teams, known as Project Aristotle. The project aimed to identify the factors that make some teams more successful than others. The study was based on the premise that the composition of a team was not the most critical factor in determining success but rather how team members interacted with each other.

Note: Before Project Aristotle, there was Project Oxygen, which looked into what traits make for a good manager. Some of the insights in this chapter were informed by the results of Project Oxygen, which I’ll talk about in detail in Chapter 4.

To determine what makes teams effective, the researchers first had to define what effectiveness means and how to measure it. They noticed that different roles had different perspectives on effectiveness. In general, whereas executives were interested in results such as sales numbers or product launches, team members thought that team culture was the key to team effectiveness. The team leaders indicated that ownership, vision, and goals were the most important measures.

Eventually, the researchers decided to study certain qualitative and quantitative factors that might impact team effectiveness, such as the following:

Team dynamics. Demographics, conflict resolution, goal setting, psychological safety

Personality traits. Extraversion, conscientiousness

Skill sets. Programming skills, client management

Researchers conducted interviews and reviewed existing survey data for 180 Google teams. They used this data to run 35 different statistical models and understand which of the many inputs collected impacted team effectiveness.

Project Aristotle identified five key dynamics that contribute to the success of software engineering teams (see Figure 1-1). These are listed next in the order of their importance:

Psychological safety

This was the most important factor identified by the researchers. It refers to the extent to which team members feel comfortable expressing their opinions and ideas without fear of retribution or criticism. Teams that have high levels of psychological safety tend to be more innovative and take more risks, which can lead to better outcomes. The researchers found that when teams feel safe, they:

Are less likely to leave the company

Are more likely to utilize the diverse ideas discussed by the team

Bring in more revenue and beat their sales targets

Tend to be rated highly on effectiveness by their leadership

Dependability

This refers to the extent to which team members can rely on each other to complete their work and meet deadlines. Teams in which individuals trust each other to be dependable are more likely to be efficient and effective in their work.

Structure and clarity

These are conditions under which team members clearly understand the project’s goals and their own individual roles and responsibilities. Team members who clearly understand what is expected of them tend to be more productive and focused.

Meaning

This refers to the extent to which team members feel that their work is meaningful and has a purpose. Teams with a strong sense of purpose tend to be more motivated and engaged.

Impact

This refers to how team members believe their work is making a difference and impacting the organization or society. Teams with a strong sense of impact are more committed to their work and the project’s success.

Figure 1-1. Google’s Project Aristotle: The five dynamics of effective teams

While Project Aristotle’s research was conducted within Google, the identified factors influencing team effectiveness could hold some relevance for teams in other contexts. By focusing on these five factors, software engineering teams can create an environment conducive to collaboration, innovation, and success. As I’ll discuss in Chapter 4, a good manager can foster these dynamics in their teams.

The researchers also discovered that variables such as team composition (size and colocation) or individual attributes (extroverted nature, seniority, tenure, etc.) did not contribute significantly to team effectiveness at Google. While these variables did not significantly impact team effectiveness measurements at Google, that doesn’t mean they’re unimportant, as indicated in the following section.

Other Research

While Project Aristotle is perhaps the best-known study on effective software engineering teams, many other studies have explored factors such as team composition, communication, leadership, and work processes. Here are a few key findings from some of these studies:

Smaller teams are better.

Although Project Aristotle did not recognize team size as relevant to team effectiveness, other studies have shown that smaller teams work better. As a team gets bigger, the number of links that need to be managed among members increases exponentially. Managing these multiple communication channels can be complicated. Many researchers have identified smaller teams containing less than 10 members as more likely to achieve success than larger teams.

Diversity can be beneficial.

It is sometimes suggested that team diversity may lead to communication and coordination problems. For example, a diverse team would usually consist of people from different family backgrounds. Those with young children are more likely to seek flexible work hours, leading to coordination challenges. However, others have found that diverse teams can be more innovative and effective. A study by Lu Hong and Scott Page of the University of Michigan found that groups of randomly selected (likely diverse) high-ability problem solvers can outperform groups comprising the best problem solvers. However, it’s important to note that diversity alone is not enough. Teams must also create an inclusive and respectful environment for all team members. For example, a team that is supportive of team members who need flexible work arrangements will be able to coordinate better than a team that is intolerant of members with such needs.

Clear communication is vital.

Effective communication is essential for effective teamwork. Studies have found that teams that communicate frequently and openly are more successful than those that do not. The idea of psychological safety is a shared belief among team members that they can freely express their thoughts, ideas, concerns, or even mistakes without fear of negative consequences or judgment. Its importance is backed up by the research from Project Aristotle. Clear communication also provides the glue to connect team members and establish structure and clarity within the team.

Leadership matters.

The leadership of a software engineering team can have a significant impact on its success. Google’s Project Oxygen showed that although teams could function without a leader, there is still a need for managers. It identified the essential traits that make for good managers and effective teams. I will talk about these traits in Chapter 4, but for now, it’s necessary to understand that there is a strong correlation between effective leadership and positive team outcomes.

Agility enables adaptability.

Agility is the ability to adapt quickly to changing circumstances. In software engineering, this means being able to pivot when requirements change or when unexpected issues arise. Agile teams are quick to adapt and can work swiftly and efficiently while maintaining high quality. A study by McKinsey & Company found that organizations that underwent successful agile transformations reported a significant improvement in efficiency, speed, customer satisfaction, innovation, and employee engagement, all of which are essential to effectiveness.

Colocation powers innovation.

The debate over whether colocation or remote work is better for software team effectiveness is ongoing, with both approaches having their own advantages and disadvantages. Multiple studies conducted at Harvard, Stanford, and others discuss the benefits of remote or hybrid work in terms of employee satisfaction and retention. However, other studies have shown that face-to-face interactions at the workplace, both planned and serendipitous, trigger the flow of knowledge, sharing of values, and exchange of ideas, which contribute to innovation.

While there may be trivial differences in the findings, we can build a theoretical picture of an ideal effective team based on the research findings discussed in this section. See Figure 1-2. By enabling psychological safety, clarity of structure and communication, dependability, meaningful work, and agility, software engineering teams can create an environment conducive to collaboration, innovation, and success.

You can now build on this understanding of dynamics and factors that influence the effectiveness of teams. The next things to consider are how the working environment can affect teams and how motivation can prime your team for success. As you go through the next sections, notice how the factors that affect teams pop up in various contexts.

3. Leadership Roles: TL, EM, TLM

The below is an excerpt from the middle of Chapter 7: Becoming an effective leader

Organizational structures in software engineering organizations differ widely depending on their culture and priorities. After a person has served as an engineer or senior engineer for a few years and gained the necessary expertise, there are typically two tracks open to them: technical or managerial. Each offers distinct leadership opportunities and requires individuals who can coach and guide their teams through challenges.

In this section, you will look at some typical roles across the industry and what they usually entail in terms of effective leadership. Note that these aren’t the only leadership roles in an organization.

Leadership roles in a team depend not only on the overall organizational structure but also on the size and complexity of the project. Larger teams could have one or many technical leads leading the development of different parts of a project. Additionally, such teams would have architects synchronize the efforts led by the technical leads and managers to plan and organize resources. You could also have a product manager who articulates what success looks like for a product and guides the team to make it a reality. Conversely, in small teams, these roles may be combined to have a manager with the technical expertise to lead the team.

Figure 7-2 shows how some of the different types of leadership roles may coexist in a software engineering team.

Figure 7-2. Relationships between various leadership roles in a software engineering team

Let’s take a closer look at some of these leadership roles.

Technical Lead

A technical lead is a hands-on role where you provide technical guidance and direction to the engineering team. The designation itself may vary across organizations. It may be a formal title in some workplaces, while it exists more informally in others. In some organizations, the position may be identified as a “software architect,” while in others, it may be referred to by titles like “principal engineer” or “lead software engineer.”

Irrespective of the name, tech leads play a crucial role in architectural decisions, code reviews, and mentoring junior team members. Technical leads often bridge the gap between the development team and management, ensuring alignment between technical strategies and business goals. Some of the responsibilities of a technical lead include the following:

Guide technical design and architecture

Tech leads play a vital role in shaping the technical direction of the project by providing guidance on design and architecture. A tech lead must leverage their expertise to ensure that the chosen technical solutions align with the project’s goals and adhere to industry best practices.

Set coding standards and best practices

Tech leads should take the initiative to establish coding standards and best practices within the development team. The tech lead role involves defining and enforcing these guidelines to contribute to overall code quality, maintainability, and consistency.

Lead troubleshooting of complex bugs and issues

Someone in the tech lead role leads the investigation and resolution of intricate technical issues and bugs. Their deep understanding of the codebase empowers them to troubleshoot effectively, ensuring the stability and reliability of the software.

Make key technical decisions with engineering trade-offs

Tech leads are responsible for making critical technical decisions, carefully weighing engineering trade-offs to align with project objectives. They consider factors such as performance, scalability, and maintainability to ensure the overall success of the software.

Do hands-on coding alongside the team

Despite their leadership role, tech leads often find themselves actively engaging in hands-on coding alongside their team members. This approach helps them mentor other engineers while staying connected with the codebase.

Serve as a mentor for development skills

Tech leads also act as overall mentors, guiding team members to enhance their development skills. They lead by example to foster a culture of continuous learning and professional development within the team.

Ensure deliverables meet the quality bar

Tech leads are accountable for the quality of deliverables, ensuring that the software meets established standards and requirements. They conduct thorough reviews and quality assessments to guarantee that the end product aligns with the defined quality bar.

Depending on the size of the project, the scope of these responsibilities will vary—from overseeing a single development team to having cross-team responsibilities.

Engineering Manager

An engineering manager typically oversees a team of software engineers, ensuring the successful delivery of projects. They are responsible for project planning, resource allocation, team productivity, performance, and career development, including that of the tech lead. This role often involves a mix of managerial tasks, such as performance evaluations and career development, along with technical oversight. In some companies, engineering managers may also be referred to as “development managers” or “technical managers.” To recap, an engineering manager’s key responsibilities include the following:

People management

Engineering managers should gear up to develop their skills in hiring, talent development, coaching, and mentoring. Engineering managers actively engage in the recruitment process, nurture their team members’ potential, provide guidance, and foster a culture of continuous learning within their team.

Manage processes

Engineering managers orchestrate critical processes such as sprint planning, retrospectives, and regular one-on-ones. They should ensure these processes are not just executed but tailored to their team’s needs, promoting collaboration, communication, and continuous improvement. They need to check that processes are not sidestepped.

Align team with organizational priorities

Engineering managers must ensure that their team is aligned with the broader organizational priorities. This involves effectively communicating context, goals, and expectations to team members while also shielding them from unnecessary distractions. By serving as a bridge between the team and the larger organization, the engineering manager helps team members focus on their work and deliver value.

Unblock resources

Engineering managers must actively work on unblocking resources needed for execution. They liaise with other departments, manage dependencies, and ensure that their team has the necessary tools, resources, and support to deliver on their commitments.

Technical oversight

While the engineering manager may not have any hands-on coding time, they should maintain their technical acumen. They engage in architecture discussions, ensuring technical decisions align with best practices and organizational goals. This technical oversight helps them guide their team to find sound technical solutions.

Stakeholder interaction

Engineering managers should engage with stakeholders, including having direct interactions with customers. They must understand project requirements, ensure proper communication channels, and act as a conduit between their team and external stakeholders. Engineering managers ensure that the team receives clear requirements from stakeholders.

Strategic work prioritization

Engineering managers must strategically prioritize work aligned with their team and company’s vision. This involves balancing project commitments with essential operational work, addressing technical debt, performing and maintenance in line with the organization’s strategy.

As you take on an engineering manager role, remember that you must broaden your responsibilities to include comprehensive people management, process leadership, and strategic alignment with organizational goals in addition to technical oversight. Unblocking your programmers is also an essential but slightly underrated aspect of managerial responsibilities.

Joel Spolsky, the cofounder of Stack Overflow and creator of Trello, once said, “Your first priority as the manager of a software team is building the development abstraction layer.”1 He further explains that if a developer is directly exposed to infrastructure issues like access to the project repo on GitHub or overriding a firewall for necessary project work, then the abstraction has failed.

Tech Lead Manager (TLM)

Tech lead managers (TLMs) are rare in many organizations. In Google, small or nascent teams usually have a TLM who can oversee a group of engineers, guiding them in project execution and ensuring the team’s productivity. This role involves a mix of technical leadership, project management, and people management. You will need a solid technical background to take up this role and should be able to contribute to technical discussions easily. You should be involved in technical design and communicate relevant design decisions to other teams and stakeholders.

TLMs are responsible for setting priorities, resolving technical challenges, and fostering a collaborative team culture. This role offers the opportunity to do both technical execution and people leadership. But it also comes with the challenge of balancing the two areas while not shortchanging either one. To help with this, TLMs will usually have a smaller number of direct reports as compared to engineering managers. TLM responsibilities include the following:

Blending people management with hands-on technical leadership

TLMs must balance their responsibilities as people manager and technical leader. This involves not only overseeing the professional development of the team but also actively participating in the technical aspects of projects, setting an example for team members.

Coach and develop engineers on coding skills

From a people management perspective, part of the TLM’s responsibility is nurturing their team, coaching, providing constructive feedback, and guiding engineers to enhance their technical proficiency. TLMs must also ensure individual contributors are challenged in their work and are on track to reach their personal career goals.

Establish technical standards and architecture

TLMs are responsible for setting technical standards and architecture. This entails defining and maintaining coding practices, architectural principles, design, and code reviews.

Help unblock developers when they are stuck

TLMs play a crucial role in unblocking developers when they encounter challenges. This involves providing technical guidance, removing impediments, and keeping upper management appraised of the project’s progress and resource needs.

Focus on higher-priority technical work

Sometimes, TLMs may need to concentrate on higher-priority technical initiatives. This could even involve hands-on coding or debugging. TLMs may have to delegate specific people management tasks to balance the other demands of their role. This strategic delegation ensures that both aspects of their role receive adequate attention.

Advocate for the team while coordinating cross-functionally

As the advocate for their team, TLMs engage in cross-functional coordination. This includes representing their team’s interests, ensuring effective communication across departments, and fostering collaboration to achieve collective goals.

Make technical decisions weighing various constraints

TLMs are decision makers in technical matters, which involves considering multiple constraints. This includes assessing factors such as project timelines, resource availability, and technical debt to make informed decisions that align with both short-term goals and long-term sustainability.

Provide mentorship and guidance

TLMs play a crucial role in mentoring and guiding team members to enhance their technical skills and professional development. By dedicating time to mentorship, TLMs foster a culture of continuous learning and growth within the team.

As you can tell from the preceding list, having really strong technical aptitude is critical in a TLM role. A TLM often asks intelligent questions and pushes the team to find answers. TLMs communicate a lot with various people, some of whom are purely technical and others of whom are business oriented. TLMs will thus have to switch their communication style constantly. A sign of success as a TLM is effectively balancing all the responsibilities while finding some extra time to write some code occasionally.

While there may be other roles or other names used to refer to these roles among software organizations, I have tried to discuss the key responsibilities of a team leader or manager in an engineering team in this section. However, responsibilities don’t dictate your ability to perform them. How do you know you have what it takes to lead your teams effectively? Find out by assessing yourself on key leadership traits in the next section.

Parting thoughts

In the preface of the book, Addy outlined who he wrote this book for:

“This book is for engineers wanting to move into leadership roles or engineering leaders who want evidence-based guidance to improve their effectiveness and that of their teams. It is a comprehensive guide to the strategies, frameworks, and best practices that I have found to be most effective in unlocking the full potential of engineering teams and driving transformative results. By sharing real-world examples, practical insights, and actionable advice, I aim to empower you with the tools and knowledge you need to become an exceptional engineering leader in your own right.

At the heart of this book lies a deep exploration of the key traits and behaviors that distinguish highly effective engineers and engineering leaders from their peers. These are the individuals who consistently deliver outstanding results, inspire their teams to reach new heights, and make a lasting impact on the projects and initiatives they lead. By understanding and embodying these characteristics, you, too, can set yourself apart and make a meaningful difference in your role.”

It’s a great time to transition into engineering leadership roles: as there are more and more in-depth resources where engineering leaders like Addy share their hard-earned experience, and way of thinking. Additionally, this book offers a peek at how effective managers at Google operate, and philosophies that are likely to be more common at Google – like the importance of physiological safety, balancing complex interpersonal dynamics, and empowering team members to take ownership of their work.

I hope you enjoyed this deepdive into a more theoretical overview of what we know about effective engineering teams, and a look at how companies like Google think about the TL, EM and TLM roles.

To read on, you can get the book (or e-book.)

And for more reading do check out some of Addy’s other books – including the free e-book titles Software Engineering: The Soft Parts, and The Developer Experience Book. You can also follow Addy on LinkedIn, where he shares learnings on software engineering several times per week. 

As related reading, see these past The Pragmatic Engineer articles:

Engineering leadership skillset overlaps: how staff engineers, EMs, PMs, TLMs and TPMs overlap in Big Tech and high-growth startups.

Engineering career paths at Big Tech and scaleups. Levels at Big Tech, the most common software engineering career paths, and what comes after making it to Staff Engineer.

ちょっと急ですが、NIST SP800-63-4 デジタルIDガイドライン第2次公開草案を読む会の第１回を8月30日午後８時より実施します。みなさま奮ってご参加ください。

（8/29現在、随時アップデート中です）

NIST SP800-63-4の第2次公開草案（2pd）が1週間前に公開されました。本日（日本時間29日午前１時〜３時）に行われたワークショップは、計画された一連のワークショップの最初のものであり、最初の公開草案以降の最も重要な変更点のいくつかを概説しています。

Introduction and Housekeeping ワークショップはNIST特別出版物800-63 改訂4第2次公開草案に関するワークショップは、セッションの録画、スライドの入手可能性、質問のためのQ&A機能の使用など、事務連絡から始まりました。[00:00] 本日の議題は以下の通りです： NIST特別出版物800-63-4の概要 このワークショップでは、デジタルアイデンティティガイドラインの第2次公開草案に焦点を当て、主要な変更点、パブリックコメント期間、コメントの提出方法について取り上げました。[02:00] このガイドラインは、連邦政府全体のデジタルアイデンティティ管理に関する基本的な要件を定めており、基本巻とA巻、B巻、C巻の4巻で構成されています。[05:00] 変更の主要な動機 主な動機には、政府サービスへの公平なアクセスの改善、新たな脅威や技術への対応、過去の実装から得られた実世界の教訓の取り込みが含まれます。[07:00] 第1次公開草案における主要な変更点 変更点には、リスク管理の刷新、生体認証要件の更新、新しい本人確認プロセス、そしてプライバシー、使いやすさ、公平性に関する考慮事項が含まれていました。[09:00] タイムラインとパブリックコメント期間 改訂プロセスのタイムラインが確認され、2022年12月の第1次公開草案の発行と2023年8月の第2次公開草案の発行が強調されました。第2次草案のパブリックコメント期間は45日間です。[12:00] Base Volumeの主要な変更 コニー・ラサール氏は、ユーザー管理型ウォレットモデルの導入（第２章）、アイデンティティリスク管理プロセスにおけるサービス「定義」ステップの追加（第３章）、継続的な評価と改善のためのメトリクスの導入、例外事項が起きたときのリドレスメカニズムについて説明しました。[16:00] 特筆すべき点として、IdP（アイデンティティプロバイダー）の変形である利用者管理型ウォレットが導入され、「発行者」がCSP（資格情報サービスプロバイダー）として捉えられるようになりました。 更新されたデジタルアイデンティティのリスク管理プロセスには、オンラインサービスの定義、初期影響評価の実施、そして継続的なリスク評価に基づいたコントロールの調整が含まれています。[20:00] 継続的な評価と改善が強調されており、推奨されるパフォーマンス指標と、問題を公平に扱うための是正措置が提示されています。[25:00] Volume A (Identity Proofing and Enrollment)の主要変更点 デイビッド・タマサク氏は、本人確認の役割と種類の更新、IAL 1（Identity Assurance Level 1）のリバランス、新しいアイデンティティ検証の経路、不正管理要件、および証拠の検証要件の更新について強調しました。[30:00] 本人確認の役割には、本人確認エージェント、信頼できる推薦者、プロセスアシスタント、申請者の参考人が含まれるようになりました。[32:00] IAL 1のバランス調整は、申請者と資格情報サービスプロバイダーにとっての摩擦の低減と選択肢の増加に焦点を当てています。[35:00] AAL 2における新しいアイデンティティ検証の経路には、生体認証を用いないオプションやデジタル証拠の検証が含まれています。[38:00] 新しい不正管理セクションには、資格情報サービスプロバイダーと依拠当事者に対する要件、必須の不正チェック、および不正の疑いがあるケースに関するコミュニケーションチャネルが含まれています。たとえば、死亡日の確認は必須になっています。[42:00] 更新された証拠検証要件には、文書認証システムのパフォーマンス指標や本人確認エージェントのトレーニングが含まれています。[45:00] Volume B: Major Changes (Authenticators and Authentication) アンディ・リーガンチッド氏は、段階的な改良、同期可能な認証器に関する新たな要件、および利用者管理のデジタルアカウントに関するガイドラインの明確化について説明しました。[50:00] 改訂されたアカウント回復セクションでは、アカウント回復プロセスの実装に関してより明確な経路とより高い柔軟性が提供されています。[55:00] パスキーのような同期可能な認証器が現在対応されており、同期ファブリック（sync fabric）に関する追加要件が設けられています。[52:00] デジタルウォレットを認証器として使用することが明確化され、保存された回復コードや信頼できる回復用連絡先など、新しいアカウント回復方法が導入されました。[57:00] Volume C (Federation and Assertions)の主要な変更点 ライアン・ガルーゾ氏は、863 Cの更新された構造、Federation Assurance Level 3（連携保証レベル3）の修正、およびプロトコルベースの例の導入について説明しました。[01:00:00] 新しい構造には、核となる共通の連携要件と、汎用IDPフェデレーションおよびユーザー管理型ウォレットフェデレーションのための別個のセクションが含まれています。[01:02:00] ウォレットモデルにおいて、ウォレットはIdPとしてモデルすることによって取り扱うことができるのでそのようにした（←コメントが受け入れられて嬉しい。） 伝統的なIDPとウォレットとの差として、マルチユーザーかシングルユーザーかで分けている。（←ここはちょっとどうかなと思う。時間の考えも入れたほうが良いのではないか。） ３つめの変更点は、Bound Authenticators の導入。連携保証レベル3には、現在、鍵保持者（Holder of Key）のアサーションとバインドされた認証器が含まれています。[01:05:00] プロトコルベースの例も追加した。OpenID ConnectやSAMLなどの連携プロトコルを実装するための高レベルな説明を提供しています。（←バックチャンネルを使わなくてもFAL2実現できるというコメントがQ&Aにあった。実際、response type = id_token ではできるはず。iGove WG でFAL2プロファイルとか作るとよいのかもしれない。）[01:08:00] パブリックコメント期間と次のステップ パブリックコメント期間は10月7日に締め切られます。コメントはメールまたはExcelスプレッドシートを使用して提出できます。最終決定までの時間は、受け取ったコメントの量によって変わります。[01:15:00] チームは一般からのフィードバックの重要性を強調し、レビュープロセスへの参加を奨励しました。[01:20:00] 特に以下の分野についてフィードバックを求めています： これが最後のパブリックコンサルテーションとなり、新年に出版が予定されています。
You can engage through the following channels: Q&Aセッション 文書の誤受入率、生体認証のパフォーマンス、パスキーの使用など、様々な質問に対応しました。[01:25:00] チームは特定の要件について説明を行い、参加者からのさらなるコメントやフィードバックを奨励しました。[01:30:00] 閉会の辞 ワークショップは、コメントの提出と今後のワークショップへの参加を促す呼びかけで締めくくられました。チームは参加者の時間とフィードバックに感謝の意を表しました。[01:35:00]

こんにちは、富士榮です。

Googleのえーじさんが公開されているパスキーのデモサイトがめちゃ便利です。

パスキーのデモにクレデンシャルの中身を解析する機能を追加しました。attestationObjectとかも中まで見えます。https://t.co/kWVU0flcfH pic.twitter.com/p6JDWZJkpf

— Eiji Kitamura / えーじ (@agektmr) August 28, 2024

こちらです。 https://try-webauthn.appspot.com/
登録した認証器がタイプなどの情報が見やすく表示されます。
この辺りでパスキーの実装を試していた時にも認証器を色々と試しては情報の解析をしたりしていたので、このサイトを使うとその辺の作業がとっても楽ですね！

How Anthropic built Artifacts

Gergely Orosz interviews five members of Anthropic about how they built Artifacts on top of Claude with a small team in just three months.

The initial prototype used Streamlit, and the biggest challenge was building a robust sandbox to run the LLM-generated code in:

We use iFrame sandboxes with full-site process isolation. This approach has gotten robust over the years. This protects users' main Claude.ai browsing session from malicious artifacts. We also use strict Content Security Policies (CSPs) to enforce limited and controlled network access.

Artifacts were launched in general availability yesterday - previously you had to turn them on as a preview feature. Alex Albert has a 14 minute demo video up on Twitter showing the different forms of content they can create, including interactive HTML apps, Markdown, HTML, SVG, Mermaid diagrams and React Components.

Tags: claude-artifacts, anthropic, claude, gergely-orosz, ai, llms, alex-albert, sandboxing, iframes, security, ai-assisted-programming

[Aaron Ross Powell]

""First, there’s what I’ve referred to in the past as the “Quillette Effect.” Because we believe our own ideas are correct (or else we wouldn’t believe them), we tend to think that people who share our ideas are correct, as well." This whole piece is worth your time."

This whole piece is worth your time: a dive into why some of Silicon Valley's leaders seem to be disappearing down an ideological morass, using AI model collapse as an analogy. These are ideas that turn to themselves again and again to infinity.

There's a lot to be said for getting out of Silicon Valley and seeing the bubble from the outside. But you've really got to do that for yourself - or have something really catastrophic do it for you.

"The problem with model collapse is, once it goes too far, it’s difficult to correct. The solution to model collapse is to train on better data. But accomplishing that, and undoing the rapidly radicalizing right-wing ideology of these titans of the Valley, means undoing the structural causes of that self-referential and self-reinforcing cascade. And that’s no easy task."

I have no idea what would bring that about.

#Democracy

[Link]

My goal is to keep SQLite relevant and viable through the year 2050. That's a long time from now. If I knew that standard SQL was not going to change any between now and then, I'd go ahead and make non-standard extensions that allowed for FROM-clause-first queries, as that seems like a useful extension. The problem is that standard SQL will not remain static. Probably some future version of "standard SQL" will support some kind of FROM-clause-first query format. I need to ensure that whatever SQLite supports will be compatible with the standard, whenever it drops. And the only way to do that is to support nothing until after the standard appears.

When will that happen? A month? A year? Ten years? Who knows.

I'll probably take my cue from PostgreSQL. If PostgreSQL adds support for FROM-clause-first queries, then I'll do the same with SQLite, copying the PostgreSQL syntax. Until then, I'm afraid you are stuck with only traditional SELECT-first queries in SQLite.

— D. Richard Hipp

Tags: d-richard-hipp, sql, postgresql, sqlite

[Emily Peck at Axios]

"70% of Americans said they approved of unions, per Gallup's most recent poll, conducted in August."

This represents a giant change in American society: labor unions haven't been this popular since 1967. But at the same time, union membership is at a record low, at just 10%.

In other words, Americans want unions but aren't typically members. We're likely to see more and more union organization attempts over the next few years, and workplaces that are unionized may have competitive advantages over workplaces that aren't in terms of attracting workers.

Because unions have been so suppressed, managers likely also need a refresher (or a from-scratch lesson) in terms of what is legal and illegal when it comes to dealing with unions in the workplace.

Bottom line: they're not going away. And likely quite the opposite. Whatever your position on unions (I think they're an important force for worker rights), they are going to increasingly be a part of the organizational landscape.

#Labor

[Link]

Some things I’ve learned about me and writing recently:

I’m impossibly distractible. It’s a learned behavior: I check all my social networks, take a look at my email, fall down Wikipedia rabbit holes. Writing on the iPad seems to help me a lot. Those things are there too, but they feel relatively inaccessible: I don’t have a Threads app, for example, and using it on the web on that device feels like a chore. I know, I know: those things do work fine on an iPad, but shhhh, I’m getting a lot of mileage out of convincing myself that they don’t. Tiny goals help. I started using Todoist earlier this year, which is the first to-do list app that fits with the way my brain works. I have a lot of things I need to keep track of, and it’s been a huge relief across work and my life to have a list that I can keep referring to. These days, my Todoist “today” list drives my day as much as my calendar does. So it was easy to add a daily recurring “Write some novel words” task. I get to check it off if I make any progress at all; the trick is that once I start making some progress, momentum usually keeps me going until I’ve written a meaningful amount. I’ve even started logging supplementary tasks if I have a thematic idea that I want to experiment with later (today’s is a scene transition that I want to play with). I’ve got to make do with the late evenings. Between taking a toddler to and from daycare, working at ProPublica, and dealing with everything I need to in the house, the only real time I have to make progress is late at night once everyone else has gone to bed. I’m exhausted by that time of night, but to my surprise, this routine has been effective for me: I settle in the living room with my iPad, and off I go. The ergonomics of slouching on my sofa with a tablet balanced across my knees are horrible, though. I can’t stay completely serious. It turns out that I’m most motivated by my sense of humor. I tried to write a serious book, I really did, but the ironies and observations kept coming, and what I’ve wound up with is a serious topic and what I hope is a gripping plot, wrapped up in irony and a delight in poking at incongruities. Hopefully readers will find it more fun than self-indulgent; I’m having fun with it, and I hope they do too. When I have written more earnestly, I come back to my draft and instantly hate it. There’s detail in irony; it reveals truths that writing point-blank seems to miss. Not a single soul will get to see this until I have polished it within an inch of its life. I got a plot suggestion from a writing tutor and it set me back six to nine months. The suggestion was good, but it meant reworking what I’d done so far. I lost momentum on the first draft and found myself stuck in editing mode, working on the same chapters again and again. Lesson learned. We can make substantive changes later, once the whole thing is committed to the page. It’s not blogging. I’ve been blogging since 1998. Although I can always use proofreading and an editing pass, this muscle is fully-developed for me. I feel very little cognitive barrier to getting a blog post on the page, and I feel like I can do it quickly. Writing a book, on the other hand, requires much more craft: it’s like chiseling a story out of rock. I didn’t study this, and I am not a great sculptor. I wrote a lot more fiction when I was younger but dismissed it as a career path, even though it's where my heart truly lay. Only recently have I given myself permission to treat it as important. I’m under no illusions that I’m good at it, but I’m going to try anyway, because here’s what keeps me going: I love it. That’s what matters most, in a way. I love making something substantial, and I love being in a creative flow state. I’m often cackling at ideas as I furiously write them down. I’m petrified of sharing what I’ve done later on, but I’m putting that out of my mind. For now, it doesn’t matter. For now, I’m just telling myself a story, and I’m enjoying it a great deal. What happens to it afterwards is a story for another time.

In response to a post on X about China's social credit system, Paul Conlon said:

Digital ID is ultimately about access control where those who impose the system are the ones determining what you are required to be and do.

Provision of resources and liberties become conditional upon the whims of the affluent. Doesn't sound safe or convenient to me.

From X
Referenced 2024-08-28T08:10:31-0400

How Paul said this struck me because I've been thinking a lot about access control lately. I believe that we build identity systems to manage relationships, but, as Paul points out, the ultimately utility of identity systems in many cases is access control.

This isn't, by itself, a bad thing. I'm glad that Google controls access to my GMail account so that only I can use it. But it doesn't stop there. If I use my Google account to log into other things, then Google ultimately controls my access to everything I've used it for. This is federation's original sin1.

Paul's comment points out the primary problem with how we build identity systems today: when access control is centralized, it inherently shifts power towards those who manage the system. This dynamic can lead to a situation where individuals must conform to the expectations or demands of those in control, just to maintain their access to essential services or resources. While we often accept this trade-off for convenience—like using Google to manage multiple logins—the broader implications are troubling.

The more we rely on federated identity systems, with their tendency to centralization, the more we risk ceding control over our digital lives, reducing our autonomy and increasing our dependence on entities whose goals may not align with our own. This is why the principles of self-sovereign identity (SSI) are so compelling. SSI proposes a model where individuals maintain control over their own identity, reducing the risks associated with centralized access control and enhancing personal freedom in the digital realm.

Critics of SSI will claim that giving people control over their identity means we have to accept their self assertions. Nothing could be further from the truth. When someone wants me to prove I'm over 18, I use a driver's license. The state is asserting my age, not me. But I'm in control of who I show that to and where. Sovereignty is about borders and imposes a system of relationships.

Now, China could use SSI to build the social credit system. One credential, controlled by the state, that is used to access everything. SSI makes individual control structurally possible, but can’t guarantee it. Technology alone can't solve this problem. As a society, we have to want a digital world, modeled on the physical one, where individuals are the locus of control and use information and assertions from a variety of credentials to build and and interact in peer-to-peer relationships. Until we value freedom and independence in the digital world, we will yield up control of our digital lives to others who will act in their own interests, not ours.

Notes

For similar reasons, I think federated social media systems are a bad idea too, but that's another blog post.

Photo Credit: Papers Please from DALL-E (public domain). Prompt: Draw a rectangular picture of police checking identity papers of people on the street

Cerebras Inference: AI at Instant Speed

New hosted API for Llama running at absurdly high speeds: "1,800 tokens per second for Llama3.1 8B and 450 tokens per second for Llama3.1 70B".

How are they running so fast? Custom hardware. Their WSE-3 is 57x physically larger than an NVIDIA H100, and has 4 trillion transistors, 900,000 cores and 44GB of memory all on one enormous chip.

Their live chat demo just returned me a response at 1,833 tokens/second. Their API currently has a waitlist.

Via Hacker News

Tags: performance, generative-ai, llama, ai, llms

Remember how much fun it was making up plays on the sandlot?  Ever try fooling the opposing team with a trick play — a hidden…

The post Do HIDDEN listings trick homebuyers & sellers into double paydays, deny consumer savings? first appeared on Real Estate Cafe.

System prompt for val.town/townie

Val Town (previously) provides hosting and a web-based coding environment for Vals - snippets of JavaScript/TypeScript that can run server-side as scripts, on a schedule or hosting a web service.

Townie is Val's new AI bot, providing a conversational chat interface for creating fullstack web apps (with blob or SQLite persistence) as Vals.

In the most recent release of Townie Val added the ability to inspect and edit its system prompt!

I've archived a copy in this Gist, as a snapshot of how Townie works today. It's surprisingly short, relying heavily on the model's existing knowledge of Deno and TypeScript.

I enjoyed the use of "tastefully" in this bit:

Tastefully add a view source link back to the user's val if there's a natural spot for it and it fits in the context of what they're building. You can generate the val source url via import.meta.url.replace("esm.town", "val.town").

The prompt includes a few code samples, like this one demonstrating how to use Val's SQLite package:

import { sqlite } from "https://esm.town/v/stevekrouse/sqlite"; let KEY = new URL(import.meta.url).pathname.split("/").at(-1); (await sqlite.execute(`select * from ${KEY}_users where id = ?`, [1])).rows[0].id

It also reveals the existence of Val's very own delightfully simple image generation endpoint Val, currently powered by Stable Diffusion XL Lightning on fal.ai.

If you want an AI generated image, use https://maxm-imggenurl.web.val.run/the-description-of-your-image to dynamically generate one.

Here's a fun colorful raccoon with a wildly inappropriate hat.

Val are also running their own gpt-4o-mini proxy, free to users of their platform:

import { OpenAI } from "https://esm.town/v/std/openai"; const openai = new OpenAI(); const completion = await openai.chat.completions.create({ messages: [ { role: "user", content: "Say hello in a creative way" }, ], model: "gpt-4o-mini", max_tokens: 30, });

Val developer JP Posma wrote a lot more about Townie in How we built Townie – an app that generates fullstack apps, describing their prototyping process and revealing that the current model it's using is Claude 3.5 Sonnet.

Their current system prompt was refined over many different versions - initially they were including 50 example Vals at quite a high token cost, but they were able to reduce that down to the linked system prompt which includes condensed documentation and just one templated example.

Via @stevekrouse

Tags: claude-3-5-sonnet, deno, sqlite, anthropic, claude, typescript, ai, llms, prompt-engineering, ai-assisted-programming, javascript, generative-ai, val-town

Debate over “open source AI” term brings new push to formalize definition

Benj Edwards reports on the latest draft (v0.0.9) of a definition for "Open Source AI" from the Open Source Initiative.

It's been under active development for around a year now, and I think the definition is looking pretty solid. It starts by emphasizing the key values that make an AI system "open source":

An Open Source AI is an AI system made available under terms and in a way that grant the freedoms to:

Use the system for any purpose and without having to ask for permission. Study how the system works and inspect its components. Modify the system for any purpose, including to change its output. Share the system for others to use with or without modifications, for any purpose.

These freedoms apply both to a fully functional system and to discrete elements of a system. A precondition to exercising these freedoms is to have access to the preferred form to make modifications to the system.

There is one very notable absence from the definition: while it requires the code and weights be released under an OSI-approved license, the training data itself is exempt from that requirement.

At first impression this is disappointing, but I think it it's a pragmatic decision. We still haven't seen a model trained entirely on openly licensed data that's anywhere near the same class as the current batch of open weight models, all of which incorporate crawled web data or other proprietary sources.

For the OSI definition to be relevant, it needs to acknowledge this unfortunate reality of how these models are trained. Without that, we risk having a definition of "Open Source AI" that none of the currently popular models can use!

Instead of requiring the training information, the definition calls for "data information" described like this:

Data information: Sufficiently detailed information about the data used to train the system, so that a skilled person can recreate a substantially equivalent system using the same or similar data. Data information shall be made available with licenses that comply with the Open Source Definition.

The OSI's FAQ that accompanies the draft further expands on their reasoning:

Training data is valuable to study AI systems: to understand the biases that have been learned and that can impact system behavior. But training data is not part of the preferred form for making modifications to an existing AI system. The insights and correlations in that data have already been learned.

Data can be hard to share. Laws that permit training on data often limit the resharing of that same data to protect copyright or other interests. Privacy rules also give a person the rightful ability to control their most sensitive information – like decisions about their health. Similarly, much of the world’s Indigenous knowledge is protected through mechanisms that are not compatible with later-developed frameworks for rights exclusivity and sharing.

Tags: open-source, benj-edwards, generative-ai, training-data, ai

At DevRev, we are building an API-first dev-centric CRM that leverages data, design, and machine intelligence to empower developers (dev) to build, support, and grow their customers (revs), in the era of product-led growth. This blog post shares some insights on how we built routing at the edge in our multi-regional deployment.

The DevRev platform was initially deployed only in the US-East region. All data related to organizations (or tenants) and their customers (workspaces) was stored in the US-East region. This raised two concerns, primarily regarding data residency and latency, which forced us to launch our 1st region outside the US, in Mumbai (ap-south-1), in March 2024, and now we are in the process of expanding our presence into multiple other regions.

The DevRev platform is built with 250+ microservices running on Kubernetes. The entire multi-regional deployment was a massive collaborative effort between the infrastructure team and various engineering teams. We had to make many design decisions, brainstorm them, try them out, and then revisit and refine them. We’ll probably have multiple blog posts that delve deeper into such design decisions, and in this blog post, we narrow down the scope only for routing.

In general, building multi-regional routing involves designing a system that can efficiently route traffic across multiple geographic regions to optimize performance, reliability, and compliance with data residency requirements. For example, if performance is the primary concern, regardless of the user’s location, the requests from that user should be routed to the region with the lowest network latency relative to their location. However, this approach would require making business data available in multiple regions, which helps reduce latency but is not the primary goal of our multi-regional deployment. We are using Fastly as our CDN, and our approach to reducing latency is to cache the relevant business data at the corresponding Fastly PoPs closer to the user.

The primary goal of DevRev multi-regional deployment is data residency. Irrespective of the location of the user, the requests from that user should be routed to the home region. The home region is the region that hosts the organization that the user belongs to. A given organization is pinned to one region, which is a decision the user makes at the time they create it.

Before we delve deeper into the routing design, let’s take a quick look at the DevRev high-level architecture. As shown in the following diagram, all the requests initiated from DevRev API clients are intercepted at the edge, by the corresponding Fastly PoP. All the API requests are first validated by a Compute service deployed in Fastly, at the edge. This will reject any unauthenticated requests at the edge itself, so the DevRev services only need to worry about the authenticated traffic.

The second level of enforcement is at the DevRev API Gateway. All the traffic to the DevRev platform flows through the API Gateway, and it centrally enforces authentication with JWT verification. The third level of enforcement happens at the service itself. We have implemented the Service Mesh pattern with Istio; and each Envoy proxy, which sits with the service itself, in the same Kubernetes pod, verifies the JWT that comes with inbound traffic.

Multi-regional routing logic is built into the Compute service running on Fastly. This is the same service that does request validation (along with JWT verification) at the edge.

An API can be invoked by an authenticated user or it can be an anonymous request. For example, when you access the DevRev web app, some static resources are loaded before authentication. Also, some generic feature flags are loaded with no authentication, and those won’t change by the user. Another example is the https://api.devrev.ai/dev-orgs.public-info.get?slug=your_slug. The web app uses this API to discover metadata related to your organization to initiate login.

Apart from the authenticated and anonymous requests, there is also another type of request that does not carry a JWT, but some sort of a hint to indicate the region it needs to be routed to. For example, the web socket upgrade call carries an ID in the query string that indicates the region it belongs to. In the same way, the dev-orgs.create API carries a region hint in an HTTP header. Apart from the query string and the HTTP request headers, we also found some APIs that carry the region hint in the request body.

In summary, the routing at the edge has to deal with three types of requests: the requests that carry a trusted JWT, the requests that carry a region hint in the query string, the request header or the body, and the requests that carry neither a region hint nor a JWT.

The following diagram shows what we built at a high level. At the edge (the Compute service) for the above three types of requests, we find the region and then dispatch the request to the corresponding backend. This happens at the Fastly PoP closer to the user, and the Fastly Compute service is aware of all the backends (or the regions) available to dispatch requests.

The Compute service running in Fastly is an edge gateway that intercepts all the requests. This is a module written in Rust, which compiles into a WebAssembly and then runs as a Compute service in Fastly.

In the Compute service code, we have a router that handles a given domain. For example, the api.devrev.ai domain has its own router, and the app.devrev.ai domain has its own. Each router defines what needs to be validated in the request path, and a given router is attached to one backend. For example, the router behind the api.devreva.ai domain is attached to the AWS global accelerator for us-east-1 and sets the hostname to api.devrev.ai.

However, this router architecture was not sufficient to support multi-regional routing at the edge. Now, we need to find the backend attached to a router in the runtime based on the request context. In other words, we need to resolve the backend based on the request context. To support this model, we introduced a new construct called, resolver. A resolver is registered against a path, and it knows how to find the corresponding region based on the request context. If there is no resolver, then the default resolver is used, which finds the corresponding region based on a claim in the JWT. The default resolver can only be used for authenticated requests. All the unauthenticated requests (if allowed), are routed to the default region.

Congratulations on making it this far! You’ve probably noticed that there’s a missing piece. In the previous section, we mentioned that authenticated requests (which carry a JWT) are dispatched to the backend based on a claim found in the JWT. But, who adds this claim to the JWT? Let’s solve that mystery.

As shown in the first diagram, the DevRev web app uses a trusted identity provider, Auth0. Auth0 does not store any user credentials; it simply acts as an identity bus. When a user creates an organization in DevRev, we internally use the Auth0 API to create the corresponding Auth0 organization and update the metadata with the unique organization ID generated by DevRev. This unique organization ID includes the corresponding regional identifier.

When a user logs into their DevRev organization via Auth0 (after being federated through their own identity provider or a social identity provider), an action within Auth0 reads the corresponding organization ID and adds it to the token it generates. The Compute service at the edge then reads this claim, determines the corresponding region, and dispatches the request accordingly.

By implementing multi-regional routing at the edge, we make routing decisions as close to the user as possible, directly in the request path, and in a transparent manner. Neither the client nor the user needs to be aware of the different regions.

How we built multi-regional routing at the edge @ DevRev was originally published in FACILELOGIN on Medium, where people are continuing the conversation by highlighting and responding to this story.

Gemini Chat App

Google released three new Gemini models today: improved versions of Gemini 1.5 Pro and Gemini 1.5 Flash plus a new model, Gemini 1.5 Flash-8B, which is significantly faster (and will presumably be cheaper) than the regular Flash model.

The Flash-8B model is described in the Gemini 1.5 family of models paper in section 8:

By inheriting the same core architecture, optimizations, and data mixture refinements as its larger counterpart, Flash-8B demonstrates multimodal capabilities with support for context window exceeding 1 million tokens. This unique combination of speed, quality, and capabilities represents a step function leap in the domain of single-digit billion parameter models.

While Flash-8B’s smaller form factor necessarily leads to a reduction in quality compared to Flash and 1.5 Pro, it unlocks substantial benefits, particularly in terms of high throughput and extremely low latency. This translates to affordable and timely large-scale multimodal deployments, facilitating novel use cases previously deemed infeasible due to resource constraints.

The new models are available in AI Studio, but since I built my own custom prompting tool against the Gemini CORS-enabled API the other day I figured I'd build a quick UI for these new models as well.

Building this with Claude 3.5 Sonnet took literally ten minutes from start to finish - you can see that from the timestamps in the conversation. Here's the deployed app and the finished code.

The feature I really wanted to build was streaming support. I started with this example code showing how to run streaming prompts in a Node.js application, then told Claude to figure out what the client-side code for that should look like based on a snippet from my bounding box interface hack. My starting prompt:

Build me a JavaScript app (no react) that I can use to chat with the Gemini model, using the above strategy for API key usage

I still keep hearing from people who are skeptical that AI-assisted programming like this has any value. It's honestly getting a little frustrating at this point - the gains for things like rapid prototyping are so self-evident now.

Tags: claude-3-5-sonnet, gemini, ai-assisted-programming, javascript, generative-ai, ai, llms, projects, anthropic, claude

こんにちは、富士榮です。

突然ですが、本人確認書類の電子化をする理由ってなんでしょう。

ペーパーレス プラスチックカードの発行費用の削減 機械可読によるビジネスプロセスの最適化

など、頑張れば色々と理由はつけられます。

OpenArtが書いてくれたモバイル本人確認書類。なんか変。

Perplexityにデジタル化の意義について聞いてみたらこんな回答が返ってきました。

質問）

免許証やマイナンバーカードなどの本人確認書類をデジタル化することによるメリットはなんですか？

回答）

デジタル化された本人確認書類には、以下のようなメリットがあります: 

利便性の向上

携帯の容易さ

デジタル化された本人確認書類はスマートフォンなどのデバイスに保存できるため、常に携帯することが容易になります。財布や身分証ケースを持ち歩く必要がなくなり、日常生活の利便性が向上します。 迅速な提示 必要な時にすぐに提示できるため、本人確認のプロセスが迅速化されます。デジタル形式であれば、必要な情報を素早く表示したり、関連する証明書を即座に呼び出したりすることが可能になります。

 セキュリティの強化

偽造防止 デジタル技術を活用することで、偽造や改ざんを防ぐ高度なセキュリティ機能を実装できます。電子署名や暗号化技術により、従来の紙の書類よりも安全性が向上します。

紛失リスクの低減

物理的な紛失のリスクが軽減されます。デジタル形式であれば、クラウドにバックアップを取ることができ、デバイスを紛失しても復元が容易になります。

行政手続きの効率化

オンライン申請の促進 デジタル化された本人確認書類があれば、さまざまな行政手続きをオンラインで完結させることが可能になります。例えば、マイナンバーカードを利用すれば、確定申告や各種証明書の取得がオンラインで行えるようになります。

データ連携の容易さ

デジタル化により、異なる行政サービス間でのデータ連携が容易になります。これにより、複数の手続きを一度に行ったり、必要な情報を自動的に入力したりすることが可能になり、手続きの効率が大幅に向上します。

環境負荷の低減

物理的な書類の発行や更新が不要になるため、紙の使用量が削減され、環境負荷の低減につながります。また、郵送や窓口での対面手続きが減ることで、移動に伴うCO2排出量も抑えられます。

デジタル化された本人確認書類は、利便性、セキュリティ、行政効率、環境保護など、多岐にわたるメリットをもたらします。ただし、デジタル格差への配慮や、プライバシー保護の徹底など、課題にも適切に対応していく必要があります。

なんだか、想像通りすぎますね。特に後半はデジタル化の意味が違いますね。

追加で質問してみます。

質問）

デジタル化について、デジタイゼーションとデジタライゼーションの違いの観点から説明してください。

回答）

デジタイゼーションとデジタライゼーションは、デジタル化の異なる段階を表す概念です。以下にその違いを説明します:

デジタイゼーション

デジタイゼーションは、デジタル化の最初の段階を指します。 定義 アナログ形式の情報をデジタル形式に変換する過程です。 特徴 物理的な情報や資料をデジタルデータに変換します。 業務フローの一部をデジタル化することに焦点を当てます。 具体例 紙の書類や本をスキャンして電子ファイル化する 電子契約や電子署名の導入 クラウドストレージやオンラインドキュメントによるファイル管理 QRコードやバーコードの活用

デジタライゼーション

デジタライゼーションは、デジタイゼーションの次の段階を表します。 定義 ビジネスプロセス全体をデジタル化し、新たな価値を創出することです。 特徴 組織のビジネスモデル全体を一新します。 クライアントやパートナーへのサービス提供方法を改善します。 特定の業務プロセス全体をデジタル化します。 具体例 クラウドやAIなどのテクノロジーを導入して業務の自動化や最適化を図る オンラインでの販売やマーケティングを強化する 電子書籍の販売システムの構築 動画共有プラットフォームの運営

 主な違い

対象範囲:

デジタイゼーション: 個別の情報や業務フローの一部 デジタライゼーション: ビジネスプロセス全体 目的: デジタイゼーション: 情報のデジタル化、業務効率化 デジタライゼーション: ビジネスモデルの変革、新たな価値創出 影響度: デジタイゼーション: 局所的な改善 デジタライゼーション: 組織全体の変革

デジタイゼーションとデジタライゼーションは、デジタル化の異なる段階を表す概念であり、企業のデジタルトランスフォーメーション（DX）を実現するための重要なステップとなります

こうやってみると本人確認書類のデジタル化はデジタイゼーションとデジタライゼーションが混ぜこぜで語られていそうですね。

まさに崎村さんが言うところの「西用と変法」ですな。 

というか、そもそも運転免許証のデジタル化についてのメリットは「バーに入るとき免許証を全部見せる必要ないですよね。物理免許証だと選択的開示はできないけど、デジタル化すると21歳以上であることだけを示せるようになるんですよ！」ってみなさん言ってましたよね。

カリフォルニア州のモバイル運転免許証（mDL）やっているSpruceIDのWayneもBlogで語ってますよ。

https://blog.spruceid.com/top-5-takeaways-from-getting-started-with-digital-id-a-look-inside-californias-mobile-drivers-license-program/

TBDもそんな説明をしていましたね。。。

そして彼女がパスポートを係員に渡すと、「ああ、私たちは誕生日が同じなんですね！」とか「出身地は美しい島ですよね〜」とか言われてしまいます。

調べた結果VCは「過剰な個人データを明らかにせずに、法定飲酒年齢に達していることを証明する必要があるとします。完全な ID を提示する代わりに、ベンダーに VC を提示することもできます。販売者は資格情報を年齢証明として認識し、引き換えにアルコールを提供します。」 

https://idmlab.eidentity.jp/2024/03/tbdverifiable-credentials.html

たぶん、こう言うアナログではできなかったことが、デジタル化することによってできるようになる、というのがデジタライゼーションでありイノベーションなんだろうなぁ、、と強く思います。

しかし、ちょうど先日カリフォルニア州のモバイル運転免許証がGoogle Walletに搭載できるようになった、という話も出てきていますしが、Apple WalletやGoogle Walletはスマホ搭載はするものの選択的開示はできなさそうです。

https://www.gov.ca.gov/2024/08/23/californians-can-now-add-their-mobile-drivers-license-to-google-wallet/

例のApple Walletにマイナンバーカードを搭載する件はどうなんでしょうか。。。うーん。

mDoc自体は当然選択的開示をサポートしていて、ISO 18013-5:2021の6.2 Functional requirementsをみると「The interface between the mDL and the mDL reader shall support the selective release of mDL data to an mDL reader.」とあるようにサポートすべし的な記載なんですよねぇ。Presentation要求にOpenID for Verifiable Presentationsを使う場合は属性要求にPresentation Exchangeを使うと思いますが、その場合はinput_descriptorで要求属性なんかもかけるわけで、リーダーというかVerifier側はそう言う実装になっていくんでしょうけど、大前提としてウォレット側が選択するUIを持っていないと片手落ちになってしまいます。自己主権アイデンティティとか言っていた人たちはどう思っているんでしょうか。
まぁ、全員が選択的開示をしたいわけではないので、これまで通りデジタイズされた免許証を使ったりマイナンバーカードを使う人がいてもいいとは思います。ただ、重要なのは「選択肢」だと思います。
今回Google WalletやApple Wallet「にも」搭載できるようになったカリフォルニア州の運転免許証はそもそもSpruceIDのウォレットに搭載されてきていました。つまり、選択的開示できるウォレットも選択肢として残されているわけです。
一方でマイナンバーカードは先にApple Walletへの搭載に関する情報が出てきてしまったので、今から別アプリが出てきても使う側のモチベーションを選択的開示一本で向かわせるのはなかなか難しいだろうなぁ、、と思います。
政治的に色々とあったんでしょうけど、戦略的にこの辺りの情報の出し方を考えていけたらよかったですね・・・

NousResearch/DisTrO

DisTrO stands for Distributed Training Over-The-Internet - it's "a family of low latency distributed optimizers that reduce inter-GPU communication requirements by three to four orders of magnitude".

This tweet from @NousResearch helps explain why this could be a big deal:

DisTrO can increase the resilience and robustness of training LLMs by minimizing dependency on a single entity for computation. DisTrO is one step towards a more secure and equitable environment for all participants involved in building LLMs.

Without relying on a single company to manage and control the training process, researchers and institutions can have more freedom to collaborate and experiment with new techniques, algorithms, and models.

Training large models is notoriously expensive in terms of GPUs, and most training techniques require those GPUs to be collocated due to the huge amount of information that needs to be exchanged between them during the training runs.

If DisTrO works as advertised it could enable SETI@home style collaborative training projects, where thousands of home users contribute their GPUs to a larger project.

There are more technical details in the PDF preliminary report shared by Nous Research on GitHub.

I continue to hate reading PDFs on a mobile phone, so I converted that report into GitHub Flavored Markdown (to ensure support for tables) and shared that as a Gist. I used Gemini 1.5 Pro (gemini-1.5-pro-exp-0801) in Google AI Studio with the following prompt:

Convert this PDF to github-flavored markdown, including using markdown for the tables. Leave a bold note for any figures saying they should be inserted separately.

Tags: gemini, pdf, generative-ai, ai, llms

Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

In the past two months, Anthropic has started to gain momentum among software engineers. The company released its latest language model – Claude 3.5 Sonnet – on 20 June, which works noticeably better than other large language models (LLM) for coding-related work – and gives better results than ChatGPT models, which is wowing many developers. We touched on this observation in The Pulse #101: it’s the first time a company other than OpenAI potentially leads in LLM capability.

Anthropic also released a new feature, Artifacts, alongside Claude 3.5 Sonnet. It allows the creation of things like websites (single page React or HTML), code snippets, documents, diagrams, and more, with a prompt. It’s helpful for a variety of tasks, and also fun. For example, I gave Claude this prompt:

“Create a web application that allows me to tweak the color of The Pragmatic Engineer Logo. This logo is three rectangular bars that increase in height, and are in the color red.”

The result was what I asked for: a mini-web application with an interactive color picker:

The output of the prompt. I could have a blue logo, if I wish! Play with it here, or see a remixed version. Remixing means continuing with the Artifact, evolving it with further input

Other interesting examples include creating:

A game (e.g. “Pong”) with a prompt

A multi-monitor layout designer with a few minutes of giving instructions

A venture capital liquidation preference simulator with instructions provided

An image picker and color extractor

While the feature is small, it feels like it could a leap in using LLMs for collaborative work — as every Artifact can be shared, used by others, and “remixed”. Curious to learn how this product was built, I contacted the Anthropic team. Today, we look at the reality of building this feature, and we also get a sense of how Antrophic’s engineering team operates. We cover:

From the drawing board to shipping Artifacts. A scrappy prototype demonstrated on “WIP Wednesdays” kicked off what became Artifacts.

Tech stack. Streamlit, Node.js, React, Next, Tailwind, and Claude.

Using Claude to build Artifacts faster. The team not only dogfooded Claude, but used their LLM to build software faster, including Artifacts.

Timeline and team. A tiny team built and shipped this feature in just 3 months.

AI products and security. Security is part of everything Anthropic does. An explainer on how model security works, and the product security approach for Artifacts.

When is an idea a winner? Not even the engineers building this feature expected it to be as successful as it is.

GenAI paradigm shift? Artifacts is not a massive feature in itself, but it could pave the way for GenAI becoming a much more collaborative tool.

At the time of publishing, Artifacts is enabled by default for all Claude users on web and mobile. So you can try out this feature, if you prefer.

This is an in-depth look into how a complex engineering project was built. Read other, similar, real-world engineering challenge deepdives.

1. From drawing board to shipping Artifacts

Let’s jump into how Artifacts was built. For this, I talked with five current Anthropic employees involved in its creation:

Research scientist, Alex Tamkin, who built and demoed the first prototype

Product designer, Michael Wang, who joined iteration efforts early

Product engineer, Florian Scholz, who helped productionize Artifacts

Security engineer, Ziyad Edher, who helped evaluate security for Artifacts

Brand, Sam McAllister, who created several launch materials for Artifacts

Scratching the itch to move faster 

In March 2024, research scientist Alex Tamkin was testing the website generation capabilities of Anthropic’s newest model, using these steps:

Prompt model to generate HTML code for a website

Copy generated code into an editor

Save file as an HTML

Open a web browser to view HTML file

The overhead wasn’t too bad from doing this once or twice. But Alex did it dozens of times, he recalled:

“This whole round-trip process was taking a lot of time. I kept thinking:

‘What if I could just see it right away?’

You know that feeling when you're cooking and you want to taste the sauce straight away, not wait for it to simmer? That's what I was after. I just wanted it to render on the screen immediately.”

So Alex put together a janky side-by-side interface, with Claude on the right and the realtime output on the left. He then showed this rough-around-the-edges demo to his team at a regular catchup session called “WIP Wednesday”:

Alex’s demo for WIP Wednesdays

The demo was a turning point, he says:

“I think this demo was when a lot of us realized: ‘oh wow, there's something here.’ 

Seeing it immediately on the screen, something sort of... clicks. You know how sometimes you don't realize you need something until you see it? That's what happened. 

It wasn't just about making the process faster. It was about changing how we interact with Claude, making it more tangible, more immediate, more collaborative.”

First prototype

One demo participant was product designer Michael Wang, who then helped make the rough demo into a more production-ready experience. He says:

“I just kept replaying this demo from Alex over and over again in my head. So I started building a prototype, mainly to see how much we could actually pull off with some basic prompt engineering and instructions for Claude. Turns out, quite a bit. I had a proof of concept working much faster than I expected. And it just got my mind racing. Eventually, I felt like I had a pretty solid idea, and I posted it to Slack.”

Posting to internal Slack was a great idea, as it got the attention of many colleagues, including Anthropic’s CEO, Dario Amodei, who offered encouragement. After this, things moved quickly, says Michael:

“In about a week and a half, we had it ready for internal dogfooding. The entire company could start using it. It was a bit surreal seeing something go from an idea to a tool that everyone was experimenting with in such a short time. But that's often how it goes when you're working with Claude – sometimes things just click, and you find yourself building something you didn't even know was possible a week ago.”

Michael demoing a polished version of the feature that became Artifacts

As a more polished version took shape, Michael shared the demo internally, gathering even more feedback and encouragement from colleagues:

Michael sharing an update on Artifacts/Iteration Mode on the internal Anthropic Slack New engineer helps ship Artifacts

Engineers at Anthropic have a lot of autonomy, and are expected to take advantage of it. Product engineer Florian Scholz was just getting started at the company, when he saw the demo and decided to help ship the new feature. He recalls:

“Alex's first demo of Artifacts happened in my second week at Anthropic. I was still onboarding in the San Francisco office and adjusting to a very new environment, so I put it on the back burner at the time. Later, when Michael showed a working prototype, I jumped right in. 

We all had a common realization that this feature was a step change. My immediate focus was on getting our infrastructure to a place where it was secure. We were concerned about preventing any issues that might arise from untrusted code generated by Claude. It was a pretty great introduction to the kind of fun challenges we face on product engineering at Anthropic.”

Demoing Artifacts with an 8-bit game

With the product ready to ship in beta, there was one last thing to do: create launch materials to showcase Artifacts, and how people can use it. This is where Sam McAllister came in, who leads Brand communications for Claude. After seeing the first prototype of Artifacts, he realized this feature was a truly differentiating UI layer. He’d been using Artifacts as it was built, and put together a demo to showcase the feature: generating an 8-bit game featuring a crab called “Claw’d:”

Screenshot of the “Claw’d” 2D, 8-bit style arcade game, generated by Artifacts. See how it was built in this 1-minute video

Anthropic recently shared a video on how they built Artifacts, which features interviews with research scientist Alex Tamkin and product designer Michael Wang.

2. Tech stack Prototyping phase

First version: When Alex built the early version of Artifacts – one that he showed to a few of his colleagues internally – he used Streamlit. This is a tool to turn Python data scripts into shareable web apps, quickly – to build a prototype for the team. 

Using a dedicated prototyping framework to build a “visual proof of concept” turned out to be a helpful approach. It enabled quick feedback, and served as a reminder that the prototype was not (yet) production ready. Of course, not all prototypes become production features, and frameworks that allow research scientists to showcase their ideas are useful, as this case shows.

Second version: Node.js. After getting good feedback, Alex was ready to share the feature with the whole company to try out. For this stage, he decided to migrate the backend from Streamlit. He wanted to use a technology that would work better with more broader usage. Alex explains the reasoning:

“I migrated the app to a Node.js setup and implemented a side-by-side layout for rendering, which I felt improved user experience.

We hold ‘WIP Wednesdays’ meetings at Anthropic, where we share our works in progress with the wider team. Sharing work at a WIP Wednesday like this was a really nice forcing function. I worked late the night before in the office, super focused and just jamming on the prompt and the overall interaction pattern. I paired with Michael too, and he helped me debug what ended up being a simple CORS issue that I was having trouble with. At this point, Claude 3 Opus couldn't actually fix the issue on its own.”

Frontend

The technology used to build Artifact is a common-enough frontend stack used by many web teams:

React: frontend framework used to build the interface

Next.js: a React framework with performance and developer efficiency improvements that many React teams use

Tailwind CSS: a utility-first CSS framework to design delightful user interfaces 

Using sandboxing primitives was a notable difference from how most web apps are built. Artifacts needs to isolate untrusted code in the sandbox; the team calls this approach a “secure playground.” As product engineer Florian Scholz puts it:

“This sandboxing approach gives us a clearly defined environment so we can deploy confidently. It's not a static sandbox, we're constantly pushing and expanding on its capabilities. Having this secure playground was instrumental in enabling us to ship so quickly.”

But how exactly did Anthropic build its sandbox; does it use browser sandboxing primitives like the Chrome V8 sandbox? Security engineer Ziyad Edher reveals details:

“We're not using any actual "sandbox" primitive per se. 

We use iFrame sandboxes with full-site process isolation. This approach has gotten robust over the years. This protects users' main Claude.ai browsing session from malicious artifacts. We also use strict Content Security Policies (CSPs) to enforce limited and controlled network access. 

These approaches protect user data from being exfiltrated through malicious artifacts. We're continuously working on hardening these environments as the browser ecosystem changes.”

Backend

Evolution has reduced the need for a more traditional backend, at least for something like Artifacts. Michael says:

“Our models have gotten so capable that a lot of what you'd normally build as backend logic, you can now just ask for! You give Claude the instructions, lay out the parameters, and get back exactly the structured data you need, formatted just the way you want.

A lot of people looking at Artifacts probably assume there's this incredibly complex backend system running the show. 

The reality is, a huge chunk of Artifacts is ‘just’ presentational UI. The heavy lifting is happening in the model itself. It's not that traditional backend work disappears entirely, but the balance shifts. I think we're just scratching the surface of what this approach can do. As these models continue to evolve, who knows?”

3. Using AI to build Artifacts faster

The team behind Artifacts leaned on Claude heavily to build Artifacts. Here’s how research scientist Alex Tamkin used Claude 3 Opus:

“Claude 3 Opus was, at the time, our most intelligent model. The process was straightforward: I'd describe the UI I wanted for Claude, and it would generate the code. I'd copy this code over and render it. I’d then take a look at what I liked or didn't like, spot any bugs, and just keep repeating that process. 

It was a really quick way to iterate on ideas!

When you can see something immediately on the screen, there's this moment where things just ‘click’. That's what I was aiming for with this approach – trying to get to those "a-ha!" moments faster.”

Screenshot of the multi-window iteration process to create a simple HTML5 web app with Claude 3 Opus, before Artifacts was ready

Florian Scholz, product engineer on the Artifact team, used Claude extensively, too. He says:

“Claude proved particularly useful as I went digging into the depths of obscure browser APIs and capabilities. I was using it to figure out how to implement specific interaction patterns, like configuring content security policy options, iFrame interactions, and DOM selection APIs. I used it for lots of areas where documentation can be thin or pretty complicated. 

Since the launch of Sonnet and Artifacts, I've been using them to jam on experimental versions of new features and get them up and running. Claude usually gives me a good starting point and I can then pair with Claude and iterate from there. I find these tools helpful to avoid the “blank page” problem.”

Within Anthropic, Sonnet 3.5 was seen as a “game-changer,” and pushed the Artifacts team to be more ambitious. Product designer Michael Wang, shares:

“I'm almost always using Claude in my development process. Claude has become such an integral part of my workflow that I'm honestly not sure what I'd do if I couldn't use it anymore. I use it to scaffold out my code, have ongoing conversations about implementation details, and transform code as needed.

Claude 3.5 Sonnet wasn't ready to test during the initial prototyping phases of Artifacts. So at the time, I was primarily using Claude 3 Opus. 

When we got an early peek at 3.5 Sonnet, it was a game-changer. Internally, folks were demoing entire Three.js or WebGL apps created by Sonnet in one shot. That's when I knew we could be a lot more ambitious with what we were building. Sonnet had a huge impact on our feature set in the month leading up to the launch. It really pushed us to expand what we thought was even possible with Artifacts.”

4. Timeline and team

Artifacts is one of the most talked-about releases from Anthropic this year, in software engineering circles, anyway! I asked product design engineer Michael Wang about the team size and timeline, from an idea all the way to production. This is how it played out:

“After Alex’s demo, I started working on the prototype on the main claude.ai repository on March 21 2024. 

There was one person working on it full time, another part-time contributing on a regular basis. We had a few other helpful hands contributing at strategic points, and a bunch of other Anthropic employees dogfooding along the way. 

The project shipped 3 months after the first demo on June 20. We shipped Artifacts alongside our most capable model yet, Claude 3.5 Sonnet.

The whole project felt kind of like a scrappy operation. But that's how some of the best stuff comes together, right?”

Previously in The Pragmatic Engineer, we covered small teams shipping impactful products – such as the dozen engineers who shipped Meta’s Threads app in 6 months. Still, Artifacts might be the scrappiest, high-impact product I’ve encountered! Congrats to everyone at Anthropic who helped build it.

Engineering, design, and research collaboration

Read more

Everyone alive today has grown up in a world where you can’t believe everything you read. Now we need to adapt to a world where that applies just as equally to photos and videos. Trusting the sources of what we believe is becoming more important than ever.

— John Gruber

Tags: generative-ai, ai, ethics, john-gruber

[Heather Bryant]

Arguing that it's harder to just be a human online, Heather Bryant has published an online participation disclaimer:

"The following disclaimer applies to participation in discourse as it relates to my individual experience as a human being in a global online community and the collective communication occurring therein. This disclaimer is intended to acknowledge the complexities, challenges and sometimes human incompatibility with discourse occurring at potentially global scale."

Honestly, this disclaimer feels universal: it's something that I would feel comfortable posting on my own site or linking to. It's both very complete and a little bit sad: these things should be commonly understood. In some ways, these clauses are obvious. But by naming them, Heather is making a statement about what it means to participate in online discourse, and what the experience of that actually is for her.

It's worth reflecting on everything here, but in particular the "some things for some people" and "spheres of relevance" sections hit home for me. It's a commonly-held nerd fallacy (forgive me for using that term) that everything is for everyone, and that everything is relevant for comment. The conversational equivalent of inviting people from multiple facets of your life to the same party and assuming it'll all go great.

It's worth asking: if you had such a disclaimer, would it be any different? What do you wish was commonly understood?

#Media

[Link]

[tante]

Tante responds to Amazon's claim that using its internal AI for coding saved 4500 person years of work:

"Amazon wants to present themselves as AI company and platform. So of course their promises of gains are always advertising for their platform and tools. Advertising might have a tendency to exaggerate. A bit. Maybe. So I heard."

He makes solid points here about maintenance costs given the inevitably lower-quality code, and intangibles like the brain drain effect on the team over time. And, of course, he's right to warn that something that works for a company the size of Amazon will not necessarily (and in fact probably won't) make sense for smaller organizations.

As he points out:

"It’s the whole “we need to run Microservices and Kubernetes because Amazon and Google do similar things” thing again when that’s a categorically different problem pace than what most companies have to deal with."

Right.

#AI

[Link]

こんにちは、富士榮です。

OpenID Foundationや関係団体（Cloud Signature Consortium、GLEIF、IIF、OIX、その他関係する機関や個人）はグローバルで相互運用可能なアイデンティティ保証に関するネットワークに根ざしたエコシステムの達成を目指してGAIN（Global Assured Identity Network）の名のもと活動を行ってきました。

2023年、OpenID FoundationはGAIN POC Community Groupを組成して、主にOpenID Connect for Identity AssuranceとOpenID Federationをテクノロジースタックとしてどのように適用できるかについて検討を進めてきました。

その結果がGAIN in 2023というホワイトペーパーとして発行されています。

https://openid.net/announcing-gain-in-2023-whitepaper/

もちろん先に掲げた大きな目標については一朝一夕で達成されるわけではないため、継続的な議論や技術研究が必要になるわけですが、最近OpenID Foundationから2024年の取り組みについて発表がありました。

GAIN Community Group: An Update

https://openid.net/gain-community-update/

発表によると、2024年はOpenID for Verifiable Credential Issuanceにフォーカスしており、既にMeeco、Talao、Datevのウォレットを使ってIETF SD-JWT VCを発行できることの確認ができています。

今後はOpenID for Verifiable Presentationsへの展開、Open Identity Exchangeとの協業によるOpenID Federationとの組み合わせでのIssuerの信頼性の確立に向けた取り組みが続けられるということですので引き続き要注目ですね。

[The 19th]

"To understand how the anti-trans agenda could reshape all of our lives, The 19th set out to examine how the laws and rhetoric behind it are impacting Americans."

My friends at The 19th dive into how the wave of anti-trans legislation and rhetoric is impacting American life.

This is a vital conversation: 177 anti-transgender bills have become law since 2021. The country has been swept into a red wave of bigotry.

These laws have implications for everyone. As The 19th describes its rationale behind this series:

"To understand how the anti-trans agenda could reshape all of our lives, our reporters have set out to examine how anti-trans laws are impacting the lives of Americans, whether or not they are trans. The goal is to connect the dots that will show how these laws, intended to target a small minority, are rewriting the future for all of us, and for generations to come. This is the Toll of America’s Anti-Trans War."

#Democracy

[Link]

[Ingrid Melander and Guy Faulconbridge at Reuters]

"[Telegram founder] Durov, who has dual French and United Arab Emirates citizenship, was arrested as part of a preliminary police investigation into allegedly allowing a wide range of crimes due to a lack of moderators on Telegram and a lack of cooperation with police."

At face value, this seems like an enormous deal: the idea that a social network operator should be arrested for not moderating and not cooperating with the police seems like a precedent with implications for a great many platforms.

Telegram has been blocked in Russia since 2018. While it's unlikely to be blocked as such in the EU, it's plausible to see a world where it's removed from app stores and made harder to access.

Decentralized platform builders in particular will be watching this carefully: what does this mean for people who are building censorship-resistant and governance-free platforms overall?

Of course, at the same time, we may not have all the information yet. We'll have to watch and see.

#Technology

[Link]

昨日、互いに独立な２方向から、「プレゼン作るなら gamma.app」という情報が流れてきた。

なので、まずは無料バージョンでやってみましたよ。

結論から言うと、かなりすごい。

アウトプットも日本語サポートしている。これが何より良い。コロナ前からわたしは beautiful.ai にはサブスクライブしているのだが、これは英語しかできない。

さらに、ベータ版だが、URLからインポートしてスライドにしてくれるのも良い。わたしのように大量に文章をブログなどで書いている人間にはうってつけだ。とりあえず、無料版で先日のディープフェイクとアイデンティティに関するブログ記事を読み込ませて作ってみたのがこれだ。一切調整する前でこのレベルであることに留意。実際には、この後、おかしいところなどを編集することになる。

Gamma.appとは？

Gamma.appは、プレゼンテーション、Webサイト、ドキュメントを簡単に作成できるオンラインツールだ。

主な特徴 多様な出力形式: プレゼンテーションだけでなく、WebサイトやドキュメントもGamma.appで作成可能 イメージやレイアウトの自動生成: 内容に合わせたイメージを自動で生成してくれる（ただし、作成されたイメージ内の文字列などはいい加減なので似た感じで作り直す必要はありそう） URLから元コンテンツをインポート可能: テキスト貼り付け、プロンプトによるAI作成の他に、ブログ記事などのURLを貼り付けて内容をインポートすることも可能だ テンプレートのインポート: 既存のPowerPointファイルをインポートし、カラースキームなどをそれに合わせて調整可能 PowerPointとの互換性: 作成したプレゼンをPowerPointにエクスポートし、既存のテンプレートを適用することも可能 多言語対応: 日本語を含む様々な言語で出力可能 使い方のヒント

Genspark や Perplexity で調査したあと、これを claude でブログ化。このブログURLをGamma.appに読み込んで1プレゼンを作成した後、PowerPointにエクスポートしてお気に入りのテンプレートを適用する2。これにより、Gamma.appの革新的な機能と、既存のPowerPointテンプレートのブランドイメージを両立できる。

料金プラン

Gamma.appには3つの料金プランがある：

無料プラン: 基本機能を試すのに最適 Plusプラン: 月額1200円（年間契約時） 10,000 AIトークン 15カードまで作成可能 Proプラン: 月額2250円（年間契約時） 25,000 AIトークン 30カードまで作成可能 カスタムドメインでの公開が可能 まとめ

Gamma.appは、日本語でもプレゼンテーション良い感じに作ってくれる、かなり可能性を秘めたツールである。AIを活用した機能、多様な出力形式、既存ツールとの互換性など、多くの魅力的な特徴を持っている。

プレゼンテーションを含むコンテンツを効率的に作成したい人には、ぜひ一度試してもらいたいツールだ。無料プランもあるので、気軽に始められるのも魅力的だ。

Gamma.appで次世代のプレゼンテーション作成体験をしてみてはどうだろうか？

こんにちは、富士榮です。

少し前にPublic ReviewとVoteが行われていたShared SignalsのImplementer's Draftが承認されています。

OpenID Foundationからのアナウンス

https://openid.net/three-shared-signals-implementers-drafts-approved/

こちらのポストでレビューについては書きましたね。

https://idmlab.eidentity.jp/2024/06/shared-signalspublic-review.html

今回承認されたImplementer's Draftは以下の通りです。

OpenID Shared Signals Framework Specification 1.0: https://openid.net/specs/openid-sharedsignals-framework-1_0-ID3.html

OpenID Continuous Access Evaluation Profile 1.0: https://openid.net/specs/openid-caep-1_0-ID2.html

CAEP Interoperability Profile 1.0: https://openid.net/specs/openid-caep-interoperability-profile-1_0-ID1.html

なお、Shared Signals Framework（SSF）については5月に開催したOpenID TechnightのTom Satoさんの資料が素晴らしいのでぜひ見ておいてください。

https://openid.connpass.com/event/316748/

[Martin SFP Bryant at The New Stack]

"How do you get started if you want to integrate your own software with ActivityPub? [Evan] Prodromou has written a new book on this very topic, and we caught up with him to explore the practicalities of linking up with the fediverse."

I'm convinced that ActivityPub is the underlying standard that all future social software will be built on. Evan is one of the founding parents of the fediverse, and this article is a great overview. His new book will be an invaluable resource for everyone who wants to embark upon this journey.

#Fediverse

[Link]

On the right is the high school yearbook picture of Allan John Ontko, one of my best friends during the three years we were classmates at what I half-jokingly call a Lutheran academic correctional institution—because that’s what it was for me. For most of the boys there, however, it was a seminary. Allan, then known to all as A.J. was one of the seminarians. He was also completely obsessed with pipe organ music and production.

When that picture was taken, in May 1965, A.J. was seventeen, a senior about to graduate, and building a pipe organ in his parents’ basement. The main parts of the organ were so large that a cinder-block wall needed to be removed so the thing could be extracted. It was also hand-built and tested, with thousands of drilled holes, thick webworks of wiring, and solder joints. All the pipes, stops, keyboards, and foot pedals were scavenged or bought used.  I went with him on some of his trips around New Jersey and New York to obtain pipe organ parts. His determination to complete the project was absolute. Just as amazing was that he did this work on weekends, since ours was a boarding school in Westchester, New York, and his home was more than an hour’s drive away in New Jersey. His parents’ patience and moral support seemed infinite.

A.J. sold the organ to some church. That was the first step on his main path in life. Look up Ontko pipe organ and you’ll find plenty of links to his work on the Web. One of his companies, Ontko & Young, has fifteen listed organs in the Pipe Organ Database. His website, ontkopipeorgans.com, is partially preserved in the Internet Archive. Here’s one snapshot.

After high school, Allan (no longer A.J.) went to Westminster Choir College. In that time he continued building pipe organs, advancing his skills as an organist, and singing as well, I suppose. We both sang bass, though he could hold a tune while I could not. He had many amusing reports of his choir’s collaborations with the New York Philharmonic and did a brilliant impression of Leonard Bernstein’s conducting style. Allan could be very funny. I recall a list of fake organ stops that he and another organist friend (whom I am sure is reading this) created for a fake pipe organ design worthy of National Lampoon or The Onion. (Some stops were named after teachers. One I recall is the Roschkeflöte, named after Rev. Walter Roschke, who taught church history. Pastor Roschke’s nickname was Peach, because his blonde crew cut looked like fine fuzz.)

After that, Allan and I were only in occasional contact.  I know he married twice. I met his first wife at a party at their apartment when he still lived in New Jersey. That was soon after college. Later he divorced, moved to Charleston, South Carolina, and remarried. Most of what I know about his time in Charleston was that the organ-building continued, along with work composing and playing music.

A surprise came in 2011, when I got a friend request on Facebook from Olivia Margaret Ontko. At first, I assumed that this must be a relative of Allan’s, since he came from a large Slovak family in New Jersey, and I attended some family gatherings there. When I went to Olivia’s Facebook page, I thought for a moment that looks kind of like Allan but… holy shit: that is Allen. He was a woman now.

I accepted the request, and marveled at how well Olivia had gathered a large collection of supportive friends, and that she had become an active advocate for trans rights and acceptance of gender choices. For the next few years, we would occasionally comment on each others’ posts, and talk now and then on the phone. Olivia also created a LinkedIn account which is still there.

Our longest conversations were in June 2015, while I was driving to the 50th reunion of our high school class, and reporting on it afterward. Olivia was still in Charleston then.  She said she couldn’t afford to come and didn’t have much appetite for it anyway. Nor did the rest of the Class of ’65, except for me. I was the only one to show up at the reunion, outnumbered six-to-one by photos of dead classmates taped to one wall of the room. Fortunately, the room was not empty of people, because it was also the 50th reunion of the graduating class that had attended the junior college sharing the same campus, and which had absorbed our high school dorm and its classrooms the year after I left. Eighteen Concordia College alumni attended, including some guys who were seniors when I was a sophomore in the high school.*

Back to Olivia.

In a high, thin voice, she recounted for me how tortured she felt through all those decades as a female in a male body. She regretted not having been born at a time, like the present, when a child who knows their body is wrongly gendered can get the medical interventions required to grow up in the right one. She said she knew from a young age that she was a lesbian, because she was sexually attracted to girls as a kid, and then women as an adult. She also lamented that the term “transsexual” was not in wide circulation back in our high school years, when it mostly referred to pioneering work Johns Hopkins was doing at the time. So Allan repressed his urge to change sex until finally deciding to become Olivia. This was a deep and moving conversation because A.J. and I were so close in our high school years, yet I had no idea what he or she was going through. Finally, she told me she had written an autobiography and would send it to me, hoping I might find a publisher. I told her I would do my best. This was a promise she repeated each time we talked after that.

Two years later, I got a surprising friend request from Allan Ontko on Facebook. The page for Olivia was gone. I accepted, and got this in response to an email:

I gave up on Olivia… the surgery involved in making the full change would have posed some serious risks and would have been totally out of my budgetary means (not to mention the cost of an entirely new wardrobe)… HA!

As we used to say in SC: Call me anything except late for supper…

Allan

Not long after that, he wrote this:

It is inconcievable that I will turn 70 this September – but for the nonce I would rather be alive and kicking. I have had a few medical problems – I am developing cataracts and double vision but they aren’t bad enough to require immediate treatment; I had total joint replacement of my right shoulder about 1 1/2 years ago; my Parkinsons is very much under control since about 2 1/2 years ago I was fitted with two brain implants (DBS).

And, I have moved since last May to Spangle WA; a little town of about 238 people which is 18 miles South of Spokane.

His address was a post office box. His phone was a cell.

My last contact with Allan was in a series of audio calls over Facebook totaling about an hour on the evening of December 17, 2020. Advancing Parkinson’s had made Allan’s speech very hard to understand, and the call kept dropping because his connection was so bad. I did gather that he was in a facility, but I didn’t catch the name. Attempts to reach him after that were for naught.

A few days ago I heard from a mutual friend that some digging on the Internet brought news that Allan had died in 2022 of Covid. My own digging led me to a list of death notices in the Spokane Spokesman-Review, with this entry among them:

ONTKO, Olivia (Age 74) Passed away February 9, 2022

The same friend also told me that Allan had become a Seventh-Day Adventist. As it happens, Spangle is the home of the Upper Columbia Seventh-Day Adventist Church, and the Upper Columbia Academy, which is across the street. So I am reaching out to both.

In The Soul’s Code, James Hillman says each of us is born with as much of a destiny, calling, mission, or fate, as an acorn has within it an oak tree. He also says,

Reading life backward enables you to see how early obsessions are the sketchy preformation of behaviors now. Sometimes the peaks of early years are never surpassed. Reading backward means that growth is less the key biographical term than form, and that development only makes sense when it reveals a facet of the original image. Of course a human life advances from day to day, and regresses, and we do see different faculties develop and watch them wither. Still, the innate image of your fate holds all in the copresence of today, yesterday, and tomorrow. Your person is not a process or a development. You are that essential image that develops, if it does. As Picasso said, “I don’t develop; I am.”

For this is the nature of an image, any image. It’s all there at once. When you look at a face before you, at a scene out your window or a painting on the wall, you see a whole gestalt. All the parts present themselves simultaneously. One bit does not cause another bit or precede it in time. It doesn’t matter whether the painter put the reddish blotches in last or first, the gray streaks as afterthoughts or as originating structure or whether they are leftover lines from a prior image on that piece of canvas: What you see is exactly what you get, all at once. And the face, too; its complexion and features form a single expression, a singular image, given all at once. So, too, the image in the acorn. You are born with a character; it is given; a gift, as the old stories say, from the guardians upon your birth.

Allen/Olivia peaked when he/she was a successful organ builder. I also gather, reading between the lines of conversation and correspondence, that things went badly toward the end of that career. There were physical and mental issues: Parkinson’s and depression were just two. Perhaps finally, there was his/her singular and determined nature. You can see it in that high school photo. We—A.J.’s friends—could also see it in a photo one of us took of him when we were all playing around with shooting pictures and darkroom work. The photo was flattering. It nailed the A.J. we knew. He looked great. But he hated it, tore it up, and threw it away.

Did he do that because the photo wasn’t of his female self? For context, all I know is what Olivia told me half a century later: that being a teenage boy didn’t work for him. What worked was knowing that building pipe organs while also writing and performing music was in his/her soul’s code.

Allen/Olivia was an extremely unique, talented, and deep person, who deserves to be recognized and remembered for what he and she brought to the world.

That’s why I still want that autobiography—if it exists. I hope someone reading this might help find it. It’s a long shot, but if you have the bulls-eye, please let me know.

*The college, called Corcordia (one of many Lutheran institutions by that name) was by then a four-year college. Five years later it too was gone. The campus, on White Plains Road in Bronxville, New York, is now part of Iona University. It’s quite lovely. Check it out if you’re in the hood.

[George Hammond at the Financial Times]

"Y Combinator, the San Francisco start-up incubator that launched Airbnb, Reddit, Stripe and Coinbase, is backing a weapons company for the first time, entering a sector it has previously shunned."

Specifically, its a low-cost cruise missile startup, which the Financial Times reports would be suitable for use in a potential war between the US and China. The cruise missiles are 10x smaller and 10x cheaper than today's alternatives, but presumably still murder people.

Also from the article:

"There is “a very interesting situation where geopolitical heat and the end of zero-interest rate policies have made people become more pragmatic,” said the founder of one start-up that was in the same group of YC-funded companies as Ares. [...] “People support builders doing cool, hard stuff.”"

Very interesting indeed. Certainly, you can make money by selling weapons of war. But should you? And in what world is killing people "cool stuff"?

Silicon Valley's origins are in large part military, of course, so this shouldn't be too much of a surprise. But for a while there, in the wake of the this-is-for-everyone radical inclusion of the web (which was not a military creation), it seemed like tech was heading in a different direction. It's disappointing to see that this was ephemeral at best.

#Technology

[Link]

NIST SP800-63-4 2pd を読まなければいけないところですが、なぜかそれから逃げて、「ABAC対ReBAC：認可ポリシーの対決」というタイトルのYouTube動画を見終わったところです。せっかくなので、動画のNottaによる要約を以下に書いておきます。

YouTube動画の要約

この動画では、GabrielとAlexとDavidが、属性ベースのアクセス制御（ABAC）と関係ベースのアクセス制御（ReBAC）、別名ポリシーアズグラフについて議論しています。彼らは、きめ細かな認可に対するこれら2つのアプローチの主な違い、それぞれの利点、および潜在的なユースケースを探っています。議論では、優れた開発者体験の提供、ソフトウェア開発ライフサイクルへの認可の統合、そして顧客の需要に基づいてSaaSやCOTSベンダーがこれらのアプローチを採用する可能性についても触れています。さらに、Alphaのようなポリシー言語の将来や標準化の取り組みの可能性についても議論しています。

主要ポイント 導入と背景

ビデオはGabrielがAlexとDavidをABACとReBACの専門家として紹介することから始まります。彼らは、きめ細かな認可の概念と、リソース属性、コンテキスト、関係性などの追加の次元を考慮することで、従来の役割ベースのアクセス制御（RBAC）とどのように異なるかについて議論します。面白い試みとして、それぞれの「推し」のアプローチとは逆のアプローチの利点をアピールするところから始まります。

00:07:06 ReBAC（ポリシーアズグラフ）の利点

まず、ABAC推しのDavidが、ReBACを擁護して、認可にグラフベースのアプローチを使用することの利点を強調しています。これには、既存のツールやフレームワークの利用可能性、オープンエンドなクエリ（検索や逆クエリ評価）の実行能力、そしてポリシーの視覚的表現（理解の助けとなる）が含まれます。Alexは、グラフが分析に適しており、既存のグラフアルゴリズムを活用できると付け加えています。

00:11:50 ABAC（ポリシーアズコード）の利点

次に、ReBAC推しのAlexが、ポリシーアズコードとしても知られるABACの利点について議論します。彼は、コーディングに慣れた開発者にとって学習曲線が低い可能性があり、成熟したXACML標準に基づいていることを示唆しています。Davidは、ABACポリシーが平易な英語の要件を密接に反映できるため、理解と維持が容易になると付け加えています。

00:17:20 複雑性の管理と採用

議論は、きめ細かな認可の複雑性の管理と、SaaSやCOTSベンダーによる潜在的な採用に転じます。Gabrielは、ユーザーとリソースを粗粒度の役割やグループに分割し、その上にきめ細かなポリシーを適用することを提案しています。Davidは、OpenID財団のAuthZenワーキンググループに言及し、これが認可APIを標準化し、ベンダーによる採用を促進する可能性があると述べています。

00:51:00 開発者体験と統合

パネリストは、優れた開発者体験の提供とソフトウェア開発ライフサイクルへのシームレスな統合の重要性を強調しています。彼らは、体験を改善するための新しいポリシー言語やツールの可能性、およびノーコードソリューションへの傾向について議論しています。Davidは、Alphaポリシー言語を発展させ、潜在的に標準化する継続的な取り組みについて言及しています。

00:55:46 認可とアプリケーションロジックの区別

聴衆からの質問に答えて、Davidは認可ポリシーとアプリケーションロジックを区別するためのガイダンスを提供しています。彼は、認可ポリシーは副作用のないものであり、報告要件に焦点を当てるべきであるのに対し、アプリケーションロジックは厳密な報告ニーズなしにビジネスルールを扱うことができると示唆しています。

感想

OpenID財団におけるAuthZEN WGの活動は、現在注目を浴び始めている活動の一つです。この対談に登場する３人はこれに関わっている人々です。このように異なるアプローチを取るPDP (Policy Decision Point, ポリシー決定点) とでも、その決定については、PEP (Policy Enforcement Point, ポリシー強制点） とのにその決定を通知できるようになるべきですし、AuthZENではそのような規格を開発しています。ご興味のある方はぜひご参加を。

動画本体

こんにちは、富士榮です。
警察庁から犯収法（施行規則）の改正に関するパブコメ募集が始まっています。 期間は8月23日〜9月24日ですので、KYCとか本人確認に関与している人たちはコメントしましょう。
https://www.npa.go.jp/news/release/2024/20240823.html

中を軽く見てみると「マイナンバー法等の改正等に伴う犯収規則等の改正」となっているので、ざっくりいうと、令和6年12月2日施行のマイナンバー法の改正で 1歳に満たない人のマイナンバーカードへの顔写真が表示されなくなること 健康保険証や公務員共済組合の組合員証などが廃止されること 医療機関で電子資格確認を受けられない（ありていに行ってマイナンバーカードを持っていない人）に対して資格証明が発行されること を受けて、また他にも、 在留カード、特別永住者証明書（16歳未満への交付時） 精神障害者保護福祉手帳 外国人登録証明書（の一部） など顔写真が表示されない本人確認書類が存在すること、また 令和6年能登半島地震にかかる本人特定事項の確認方法等に関する特例の廃止 を受けて犯収法施行規則の改正を行う、という話です。
主に第七条（本人確認書類の区分）を整理するための改正ですね。
ただ、せっかくなので対面での本人確認（当人認証）を行うために必要な顔写真がない場合への対応まで踏み込んでいけるといいですね。

I love hanging out in Reeder. I subscribe to thousands of feeds, and it handles them well for me. But it does make it hard for me to prune them once I’m subscribed.

I found myself looking at this screen this afternoon:

And I thought to myself: you know what? I don’t need to be subscribed to Axios. This isn’t the kind of article I’m looking to consume on a regular basis.

So, uh, how do I unsubscribe from it?

There’s nothing here that allows me to unsubscribe from the feed while I’m looking at this post. There’s also nothing that tells me which folder it’s in, so I can go looking for the feed and unsubscribe it there. I’m actually not really sure where I filed it. And I can’t search for feeds by name. Sure, I could have a better, more organized system, but really, I could use more help.

This contextual menu also doesn’t help me:

So until I go through my subscriptions folder by folder, I’m stuck reading pieces about the Harris campaign mocking Trump, which, to be honest, I really don’t care about.

Regardless, it’s my favorite feed reader. But I come up against this issue surprisingly regularly.

This proposed image for the fediverse is good; I like it a lot.

But I don't know that the combative language on this site is helpful. The Meta iconography isn't right, I agree, but there's something off about calling them “a large corporation that is joining in as late” (sic).

For one thing, Meta is early; for another, it seems to me that we want companies to participate? I don’t think seeking ideological purity is useful (and run the risks of the movement shooting itself in the foot).

Whatever you think about Meta’s goals for participating, I do also think Meta’s presence gives the network a sort of legitimacy that it was otherwise struggling to achieve. That’s a net benefit: we must grow the network.

I also agree with the point, made by Chris Messina, Manton Reece and others, that the right phrase is the social web, not the fediverse. The web is the network.

[Tara Tarakiyee]

"I want to talk about three examples I see of cracks that are starting to form which signal big challenges in the future of OSS."

I had a knee-jerk initial reaction to this post - what open source bubble?! - but Tara Tarakiyee makes some important points here about our dependence on open source code and how that might change over time.

The through line to all of them is about money. The OSI's new "open source AI" definition is loose because AI vendors likely couldn't make money otherwise (although whether they can make money anyway is still up for debate); source-available licenses have become prevalent because it's easier to sell commercial licenses and therefore make a living building software; much open source software was precariously funded through European Commission Next Generation Internet grants, which are now evaporating.

While we can stand for pure open source values all we like, the people who build open source software need to make a living: food must go on the table and they need a roof over their heads. Ideally their compensation would extend beyond those basic necessities.

This has been the perennial problem for open source: how can it be sustainable for the people who build it? We're not launching into a post-monetary Star Trek future any time soon. In the meantime, people need to be paid for their work, or open source runs the risk of being a hobbyist-only endeavor.

People won't pay for software that they don't need to pay for. I suspect open-core, which opens the core of a software platform while monetizing high-value extensions, is the best answer we can hope for. But even that might not be realistic.

#Technology

[Link]

[Andy Jassy on LinkedIn]

Andy Jassy on using Amazon Q, the company's generative AI assistant for software development, internally:

"The average time to upgrade an application to Java 17 plummeted from what’s typically 50 developer-days to just a few hours. We estimate this has saved us the equivalent of 4,500 developer-years of work (yes, that number is crazy but, real)."

"The benefits go beyond how much effort we’ve saved developers. The upgrades have enhanced security and reduced infrastructure costs, providing an estimated $260M in annualized efficiency gains."

Of course, Amazon is enormous, and any smaller business will need to scale down those numbers and account for efficiencies that may have occurred between engineers there.

Nevertheless, these are incredible figures. The savings are obviously real, allowing engineers to focus on actual work rather than the drudgery of upgrading Java (which is something that absolutely nobody wants to spend their time doing).

We'll see more of this - and we'll begin to see more services which allow for these efficiency gains between engineers across smaller companies, startups, non-profits, and so on. The dumb companies will use this as an excuse for reductions in force; the smart ones will use it as an opportunity to accelerate their team's productivity and build stuff that really matters.

#AI

[Link]

こんにちは、富士榮です。
デジタル庁からマイナンバーカード対面本人確認アプリがリリースされましたね。
https://services.digital.go.jp/mynumbercard-check-app/

そういえばこの手のアプリって大量に出回っている感覚があったのでAppleのAppStoreを探索してみました。
いっぱい出てきた・・・（順番などに他意はありません。適当に探索したのでカバーできているわけもありません。XX Payなどアプリの中の一機能になっているものは拾っていません） アプリ名Copyright概要マイナンバーカード対面確認アプリデジタル庁デジタル庁謹製のマイナンバーカード確認アプリマイナチェッカーLEGAL PROMPT INC.券面情報の読み取りアプリマイナンバーカード読み取り練習アプリHideki Kariyaマイナンバーカードの読み取りを練習するアプリOKBマイナンバーカード本人確認アプリTOPPAN Edge Inc.大垣共立銀行の手続きに利用する本人確認アプリ北陸銀行本人確認アプリTOPPAN Edge Inc.北陸銀行の手続きに利用する本人確認アプリSpeed Letter Plus本人確認アプリTOPPAN Edge Inc.通知物電子送付サービス「Speed Letter Plus」に利用する本人確認アプリ南牧村本人確認アプリTOPPAN Edge Inc.みなみまきパスポートの手続きに利用する本人確認アプリLIQUID eKYC株式会社LiquidLiquid社が提供する汎用本人確認アプリProTechマイナンバーIC認証Showcase IncProTechマイナンバーIC認証を利用しているWebサイトで利用する認証アプリ本人確認アプリD-ConfiaDouble Standard InceKYCアプリ。足立区で使われている？JPKI利用者ソフトJ-LISJ-LIS純正のマイナンバーカードの電子証明書利用機能提供アプリJPKI暗証番号リセットJ-LISマイナンバーカードのパスワードを初期化・リセットするアプリIDリーダーOSSTech CorporationOSSTech社が提供するIDカード読み取りアプリIAM＜アイアム＞ShiftPlus IncShiftPlus社が提供する汎用本人確認アプリ電子認証マイナサインCYBERLINKS CO.,LTD.電子署名、券面情報取得を行うアプリ本人確認アプリeKYCCYBERLINKS CO.,LTD.eKYCアプリマイナトラスト電子署名CYBERLINKS CO.,LTD.電子署名アプリマイナポータルデジタル庁言わずと知れたマイナポータルアプリデジタル認証アプリデジタル庁言わずと知れたデジタル庁認証アプリマイナポイント総務省言わずと知れたマイナポイントアプリPDFにマイナンバーカードでJPKI電子署名二進合同会社PDFに署名するアプリeKYC本人確認アプリsho@99zz.neteKYCアプリ都市OS共通サービス（マイナンバーカード共通機能）Hitachi, Ltd日立の都市OS用の本人確認アプリクロスアイディxID Inc.xID社が提供する汎用本人確認アプリTRUSTDOCKTRUSTDOCK INCTrustdock社が提供する汎用本人確認アプリe-Probatio本人確認アプリNTTビジネスソリューションズNTTビジネスソリューションズが提供する汎用本人確認アプリe-NINSHO本人確認サービスNRINRIが提供するe-NINSHOサービスの本人確認アプリe-NINSHO公的個人認証アプリNRINRIが提供するe-NINSHOサービスの公的個人認証アプリ日司連公的個人認証有効性確認システム日本司法書士連合会マイナンバーカードの券面情報の取得と有効性確認を行うアプリ当事者型電子署名システム日本司法書士連合会電子署名を行うアプリGreat eKYCTREASURY INCeKYCアプリマイナPocketNTTデータマイナンバーの申告・本人確認に使うアプリジャスミーPDL本人確認サービスJASMY本人確認アプリ本人確認情報提出アプリBIPROGYスマートシティ（Dot to Dot）の本人確認アプリスマートライフパス本人確認アプリTOPPANSmart Life Passのユーザ登録時の本人確認アプリNAWABARI eKYCアプリNAWABARIネットショップの住所に使えるサービスの本人確認アプリめぶくIDmy FinTech inc汎用デジタルIDYSD公的個人認証アプリヤマトシステム開発ヤマトシステム開発が提供する証明書類Web取得サービス用の公的個人認証アプリRSSモバイルLEGAL Corporation電子署名アプリ
どうするんでしょうね。

Back in May, I announced Unoffice Hours, inspired by something Matt Webb had established with his community. Anyone could book a 30 minute meeting with me, for any reason, on a Friday. No money, no strings, no expectations.

It’s been a wonderful experiment. I’ve met a host of new people who were new to their careers; had fascinating conversations with people I’ve been a huge fan of for years; and I’ve caught up with old friends and acquaintances.

It wasn’t without its glitches: there were a few times I had to move calendar slots around because of unexpected travel. There were also vastly more people who wanted to sign up than I had slots for (a nice problem to have!), so sometimes sessions were booked months out. Towards the end, there were a few no-shows, which isn’t perfect, but also isn’t too big a deal. On balance, the conversations were meaningful, and it was absolutely worth it.

So let’s do it again.

I’ve been a 2X startup founder, early-stage investor, software engineer, engineering lead, CTO, and CEO. I’ve taught product design to teams around the world, and I’ve built large-scale end-user open source projects. I’m deep into the fediverse and care deeply about the open web and ethical tech policy. I’ve also been trained in leadership coaching.

Here are some topics it might be interesting to chat about:

Feedback on a project you’re working on (startups, software, a writing project) Following up on something I’ve written in this space Product and technology strategy in the public interest (news, education, libraries, other mission-driven organizations) The open social and indie web Fostering a collaborative organizational culture Saying hello

If you’d like to chat, book a 30-minute session here!

ちょっと今日思い立って、今年度に入ってからのデジタルアイデンティティ関連の動きの一覧を作ってみたんですが、矢継ぎ早すぎる…。自分のXの投稿とブログの投稿をみてやっているだけなんで、多分見落としも結構あるんですが、これだけでもお腹いっぱい。まさかこのペースは続かないとは思いますが…。なお、これもいれろというようなのがあれば、ぜひ教えて下さい。

以下、X は Xへのリンク、B は当ブログへのリンク、Wは外部Webへのリンクです。

4/22 NIST SP800-63B追補 発表 [X, B] 4/30 European Digital Identity Framework 公布（5/21 EU Digital Identity Regulation 施行）[X] 5/23 EU Digital Identity Architecture and Reference Framework ver.1.4 公表 [X] 6/7 行政手続における特定の個人を識別するための番号の利用等に関する法律の一部改正関係[X, B] 6/18 国民を詐欺から守るための総合対策 [X, B] 6/19 EU AMLパッケージ 公布[X, B] 6/21 デジタル庁デジタル認証アプリ[X, B] 6/27 「個人情報保護法 いわゆる3年ごと見直しに係る検討の中間整理」に関する意見募集 [W] 7/15 デジタル庁本人確認ガイドラインの改定方針令和5年度中間とりまとめ [X, B] 7/21 英国、デジタル情報およびスマートデータ法案提出へ [X, B] 7/24 OpenID Connect for Identity Assurance パブコメ開始 [X] 7/24 Fourth Implementer’s Draft of OpenID Federation 承認 [X] 7/30「子供オンライン安全法案」(KOSA) 上院通過 [X, B] 8/12 EUDIW の Implementing Act Public Comment 開始 [X, B] 8/20 マイナンバーカード対面確認アプリリリース [Xs] 8/20 Three Shared Signals Implementer’s Drafts 承認 [X] 8/21 NIST SP800-63-4 2nd Public Draft 発表 [X, B] 8/22 「犯罪による収益の移転防止に関する法律施行規則及び犯罪による収益の移転防止に関する法律施行規則及び疑わしい取引の届出における情報通信の技術の利用に関する規則の一部を改正する命令の一部を改正する命令案」に対する意見の募集 [X] 8/31 マスターカード、オンライン決済をより安全にするパスキーを使った新サービス発表 [X] 8/31 OIX閉鎖

こんにちは、富士榮です。

すでにあちこちで書き込まれているので今更私が書くこともありませんが、みんな大好きNISTのデジタルアイデンティティガイドラインであるNIST SP800-63-4 SPD（Second Public Draft）がようやく出てきました。（本当はもっと前に出るって噂でしたが）

とりあえずパブリックコメント募集は10月7日までこちらで。

https://www.nist.gov/news-events/news/2024/08/nist-releases-second-public-draft-digital-identity-guidelines-final-review

そして、早速崎村さんが概略を日本語で書いていらっしゃいます。

https://www.sakimura.org/2024/08/6224/

オフィシャルには8月29日の午前1時から公開Webinarがあります。

https://nist.zoomgov.com/meeting/register/vJItcu6oqj0jH4xWLbz350jf6VeKM9bMjWc#/registration

さて、コメントに向けて準備しましょうかね。

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. An unacceptable Microsoft Authenticator bug has been unaddressed for years; Hardware startup Nothing explains why it is going back to 5-days-a-week working from office; Ban on US noncompetes reversed; AI startup acquired for 2,000x its annual revenue; and more.

A ‘startup purge event’ has, unfortunately, arrived. A year ago, we deduced why we can expect startups to go bankrupt, en masse, starting in 2024. This event seems to be happening, with startup shutdowns rising quickly.

Sonos’ app rewrite was a YOLO release. Sonos wants to bring back its old app because the new one is so buggy: however, it cannot. A series of self-inflicted missteps are now hurting sales at the company, and could have been the reason for recent cuts.

Similarities between AI companies and telcos. AI startups have more in common with capital-heavy businesses like telecommunications companies than they do with software startups. We can expect to see some businesses with lots of capital going under thanks to intense competition, and more complex pricing structures to emerge.

What's it like to test compilers? QA engineer Alex Zakharenko tests compilers, full-time. He shares interesting details on what this is like: from how automated tests are handy in this area as well, to examples of what his day-to-day testing tasks look like.

1. Industry pulse Unacceptable Microsoft Authenticator bug gone unaddressed

The most commonly used two-factor authentication apps are Authy, Google Authenticator, Microsoft Authenticator, Duo Mobile, and LastPass Authenticator. These apps need to work reliably – or else users will be locked out of their accounts. It turns out that Microsoft’s Authenticator has a major flaw that has gone unaddressed for years, which overwrites existing accounts when adding a new account via a QR code. CSO Online shares:

“The core of the problem? Microsoft Authenticator will overwrite an account with the same username. Given the prominent use of email addresses for usernames, most users’ apps share the same username. Google Authenticator and just about every other authenticator app add the name of the issuer — such as a bank or a car company — to avoid this issue. Microsoft only uses the username.”

It’s mind-boggling to hear that Microsoft has not addressed a longstanding and known issue that creates a major headache for its users. Such a glaring bug is a reason I would avoid using Microsoft Authenticator until this issue is fixed.

This fiasco is yet another example showcasing how Microsoft keeps under-investing in security – and security tooling.

Hardware startup explains why it is going back to 5-days-a-week working from office

Read more

[The 404 Media team]

"In the last year, we learned that the technical infrastructure exists now for even non-technical journalists to build a sustainable site that can receive money from subscribers. [...] If you are a journalist reading this thinking about going out on their own: the tech is there for you to do so with very little know-how needed."

404 Media has been a new shining light in technology journalism. That it's worked out for them, having invested a thousand dollars each at the outset, is delightful.

And then there's this:

"The biggest challenge that we face is discoverability. To the extent possible, we don’t want to have to rely on social media algorithms, search engines that don’t index us properly and which are increasingly shoving AI answers into their homepages, and an internet ecosystem that is increasingly polluted by low-quality AI spam."

So to counter that, they're building community. Which just so happens to be what every single newsroom should also be doing.

Here's to another year of 404.

#Media

[Link]

Zum Auftakt der Branchenmesse Gamescom weist die Piratenpartei die Kritik des Herstellerverbands Video Games Europe an der EU-Bürgerinitiative zum Schutz von Videospielen vor Fernabschaltung durch den Hersteller zurück. Der Industrieverband bezeichnet den geforderten Weiterbetrieb vom Hersteller aufgegebener Spiele durch die Community als “unrealistisch” wegen der Kosten für Unternehmen und weil private Serverbetreiber Datenschutz und Sicherheit nicht gewährleisten könnten.

Dr. Patrick Breyer, ehemaliger Europaabgeordneter der Piratenpartei, hält dagegen:

Hersteller und EU-Kommission müssen lernen, dass Videospiele keine Konserven sind, die man einfach aus dem Regal nehmen und durch ein anderes Produkt ersetzen kann. Sie sind Teil unseres gemeinsamen digitalen Erbes und verbinden Menschen über Jahre und manchmal sogar Jahrzehnte hinweg. Deswegen ist ein Weiterbetrieb durch die Community nur folgerichtig.

Es gibt längst Spiele, die erfolgreich ehrenamtlich betrieben oder weiterbetrieben werden. Der Datenschutz gilt auch für private ehrenamtliche Betreiber. Gamer, die in eine Übertragung ihrer Daten einwilligen und über die neuen Rahmenbedingungen aufgeklärt werden, sollten nicht wegen möglicher Abstriche bevormundet und ganz ihres Lieblingsgames beraubt werden, sondern eine Wahl bekommen. Die aktuelle Situation, in der Hersteller sich gegenüber voll zahlenden Gamern die willkürliche kurzfristige Abschaltung von einem Tag auf den anderen vorbehalten, ist untragbar.

Die Bürgerinitiative richtet sich zwar speziell gegen das willkürliche Abschalten von Computerspielen durch die Hersteller, aber diese Praxis betrifft längst nicht mehr nur Spiele. Auch andere Software und digitale Geräte bis hin zu Zügen werden fernabgeschaltet. Wir Piraten wollen diesen Rechtsmissbrauch stoppen und gesetzlich den Weiterbetrieb sowie die Weiterentwicklung verwaister Software durch die Community ermöglichen. Dies ist auch im Interesse der Nachhaltigkeit.

Die Piratenpartei setzt sich mit Nachdruck für die europäische Bürgerinitiative „Stop Killing Videogames“ ein und ruft zur Unterzeichnung auf. Die Initiative zielt darauf ab, sicherzustellen, dass Videospiele nicht unspielbar werden, wenn die Unterstützung der Publisher endet. Sie hat kurz nach dem Start bereits etwa 300.000 von 1 Mio. erforderlichen Unterschriften erreicht.

„Initiativen wie ‚Stop Killing Videogames‘ sind essenziell, um die Rechte der Spielenden zu schützen und die Vielfalt der Gaming-Kultur zu bewahren. Wir Piraten fordern, dass vermeintlich nicht lukrative Spiele nicht deaktiviert, sondern der Öffentlichkeit als Open Source zur Verfügung gestellt werden und beispielsweise private Server ermöglicht werden“, erklärt Lukas Küffner, Bundesvorsitzender der Piratenpartei Deutschland.

Auf Anfrage Breyers hatte die EU-Kommission bestätigt, dass es bisher keinen Schutz vor willkürlicher Abschaltung durch Hersteller gibt.

Weitere Informationen zur Bürgerinitiative:
Info-Website
Aufzeichnung Piraten-Webinar
Piraten-Infos

こんにちは、富士榮です。

先日、大学ICT推進協議会（AXIES）の年次大会の話をしましたが、そちらに先行して（？）アイデンティティの勉強会が開催されます。

大学の認証基盤といえばSAML、Shibbolethなのですが、そろそろOpenID Connectに対応すると言われて久しいShibbolethが本当に対応するのかどうかは置いておいて、アカデミアでも使うサービスがOpenID ConnectのRelying Partyとなっているケースも増えてきており、そろそろ学術系でもOpenID Connectの勉強会をしないとね、ということでお話しさせていただこうと思います。

※そう言えばHPCIもOAuth対応しましたしね（OAuth認証になっているのはうーん、ですが）

ということでお知らせです。（申し込みサイトからの転載）

現地でお会いしましょう。

AXIES認証基盤部会 勉強会「フェデレーション、エコシステムとID管理」

AXIES認証基盤部会では、以下の要領で2024年度 勉強会を開催いたします。会場は、学術総合センター（国立情報学研究所）とオンラインのハイブリッドで開催します。

テーマは、「フェデレーション、エコシステムとID管理」です。

NII学術情報基盤オープンフォーラムでのディスカッションを受け、学認コミュニティ拡大に向けたフェデーレション技術、エコシステムに関する勉強会を開催します。また後半では、昨年の年次大会を受け、ID管理についてディスカッションを行います。

日時：2024年9月17日（火）13:30～17:00

場所：国立情報学研究所 （オンラインのハイブリッド開催）

　　　東京都千代田区一ツ橋２丁目１−２　学術総合センター 3階

参加費：無料

定員：現地80名、オンライン200名

参加申込： 参加申込URL

プログラム：（敬称略）

13:30-13:35 はじめに

13:35-14:15 OpenID Connect（調整中）

富士榮 尚寛（伊藤忠テクノソリューションズ株式会社）

14:15-14:55 Gakunin Federation（調整中）

西村 健（国立情報学研究所）

14:55-15:05 休憩

15:05-15:45 アプリカタログ（調整中）

江川 淳一（エクスジェン・ネットワークス株式会社）

15:45-16:25 ID管理（調整中）

中村 誠（東京大学）

16:25-17:00 ディスカッション

概要

近年、デジタル社会の急速な進展に伴い、オンラインでの本人確認や認証の重要性が高まっています。この度、米国国立標準技術研究所（NIST）が、デジタルアイデンティティに関するガイドライン「SP 800-63-4」の更新案(2nd Public Draft)を発表しました。この更新は、セキュリティの強化と、誰もが公平にデジタルサービスにアクセスできる環境の整備を目指しています。

今回の更新の特筆すべき点は、最新のデジタル技術への対応です。例えば、「同期可能な認証器」（パスキー）や「ユーザー管理型ウォレット」といった新しい概念が導入されています。パスキーは従来のパスワードよりも高度なセキュリティを提供し、ユーザー管理型ウォレットは各種デジタル証明書を安全に保管することができます。

一方で、NISTは従来の本人確認方法も重視しています。スマートフォンを持たない人や、デジタル証明書を持たない人々も、必要なサービスにアクセスできるよう配慮がなされています。例えば、対面での本人確認や、通常の身分証明書を持たない人のための例外処理などが詳細に規定されています。さらに、「申請者照会」という概念も導入され、身分証明書を持たない人でも、信頼できる第三者の保証があれば本人確認が可能になります。

生体認証に関しても、ガイドラインは重要な指針を示しています。生体認証の精度と個人情報保護の重要性を強調しつつ、特に公共サービスにおいては代替手段の提供を推奨しています。また、生体認証システムにエラーが発生した場合に備えて、手動での対応プロセスの整備も求めています。

このガイドライン更新案は、民間企業や advocacy グループなど、様々なステークホルダーからのフィードバックを反映しています。NISTは、不正利用防止策と公平なサービスアクセスのバランスを取ることに注力しており、今後のデジタル社会の発展に大きな影響を与えると考えられます。

デジタルアイデンティティの管理は、今後ますます重要になってくるでしょう。NISTのこの取り組みは、セキュリティと利便性、そして公平性を両立させる先進的な試みとして、世界中から注目されています。私たちも、この動向に注目しつつ、自身のデジタルアイデンティティ管理について考えを深めていく必要があるでしょう。

本文書は、10月7日までコメントを募集しています。詳細は、こちらのページからアクセス可能です。

【付録】元リリースよりのピックアップポイント

NISTのデジタルアイデンティティガイドライン更新の概要

NISTは、セキュリティとアクセシビリティを向上させるため、デジタルアイデンティティに関するガイダンス草案を更新しました。 この更新は、民間企業や advocacy グループを含む様々な関係者からのフィードバックを反映しています。 このガイドラインは、不正対策とデジタルサービスへの公平なアクセスのバランスを取ることを目指しています。

更新されたガイドラインの主な特徴

この草案には、同期可能な認証子やユーザー管理型ウォレットなど、最新のデジタル経路に関するガイダンスが含まれています。 同期可能な認証子（パスキー）は、従来のパスワードと比較してセキュリティが向上しています。これについては附属書Bに記載しています。 ユーザー管理型ウォレットは、身分証明書を含む様々なデジタル証明書を保存できます。これについては附属書Cに記載しています。

アクセシビリティと従来の本人確認方法

このガイドラインは、スマートフォンやデジタル証明書を持たない個人でもサービスにアクセスできることを保証しています。 拡張されたガイダンスには、対面での本人確認や、従来の身分証明書を持たない人のための例外処理が含まれています。 「申請者照会 (“applicant reference”)」の概念により、身分証明書を持たない人のために信頼できる個人が保証することが可能になります。

生体認証とプライバシーへの配慮

更新されたガイダンスでは、本人確認のための生体認証の使用を維持しつつ、精度とプライバシーを強調しています。 特に公共サービスシステムにおいて、生体認証方法の代替手段が推奨されています。 NISTは、生体認証システムに潜在的なエラーに対処するための手動プロセスを含めることを目指しています。

[Jess Weatherbed at The Verge]

"A federal judge has blocked the Federal Trade Commission’s ban on noncompete agreements that make it difficult for workers to join their employers’ rivals or launch competing businesses. The ruling prevents the FTC’s ban on noncompete agreements from taking effect on September 4th, though the agency could still appeal the decision."

This is such a blow. Non-competes are already illegal in California; they're inherently anti-worker and are also very clearly a hindrance to innovation.

Hopefully the FTC appeals. The judge's argument that it would cause "irreparable harm" is nonsense: any business that has to protect itself by not allowing its workforce to go work for a competitor is obviously not competitive at what it does. Instead, businesses should seek to do good work and create conditions so that workers don't want to leave in the first place. The solution is a better workplace, not legal restrictions that prevent people from finding a different employer.

#Business

[Link]

This past week I embarked upon two long car drives — from Philadelphia to Cape Cod and back again — with an almost-two year-old. He’s a remarkably good traveler who takes everything in his stride (as long as he has his puffy stickers and you agree to put Elmo on the radio from time to time).

The biggest challenge was finding places for him to eat well: restaurants that wouldn’t bat an eyelid at a toddler doing toddler things at the table, that also would have the kinds of food that he likes. (Yes, I packed snacks. But sometimes it’s time to sit down for a real meal.)

We made it work, but it was always a bit of a guessing game. Would X chain or Y mom-and-pop accommodate his needs? I found myself wishing that there was some kind of app that would just recommend somewhere to go — particularly as he could get hungry at a moment’s notice, and faffing around with restaurant reviews while I was driving was an impossibility.

So out of interest, once I returned, I asked ChatGPT for recommendations:

Find a restaurant suitable for attending with a toddler in Elkins Park, PA. Please just return the restaurant name and address; don't share any other information.

It returns:

The Creekside Co-op
7909 High School Rd, Elkins Park, PA 19027

That’s pretty good! Creekside is a local marketplace and brewery near where I live that absolutely is suitable for hanging out with a toddler.

I tried it in a few locations. In the part of Cape Cod where we were staying, it suggested the Moonakis Cafe, which is a superb choice. In the part of Oxford, England, where I grew up, it suggested The Victoria Arms — again, no complaints whatsoever. I’d eat there with my little one in a heartbeat.

So what if I want to make this a simple app, starting on the web?

It turns out that you can replace the name of the location in the query with its GPS coordinates, which simplifies matters a great deal. You can get the user’s current location via the web geolocation API: you can then plug that straight into the query, make an API call, and forward the user directly to a Google Maps view for the restaurant. Reverse geocoding APIs, which take latitude and longitude and return a human-readable name, are also available, and might be useful — but they tend to cost money, and the API call to ChatGPT is already an expense.

More refinements are possible, of course. Most crucially, it would be helpful to know if a given restaurant is actually open; on a road trip, it would also be very useful to find restaurants that are close to the freeway. I wouldn’t mind seeing a top 3 or top 5 list with the relative merits of each one. And being able to plan a trip around toddler-friendly eating spots in advance would be gold.

What’s sort of neat about the technology is that this is all actually very feasible — in an hour or two. Using AI as an engine in this way cuts out a lot of development time, not to mention the need to gather your own database. There are obviously ethical issues related to how these models were trained, which are sure to be litigated for years to come. But here we’re not creating any new content or replacing an artist: we’re making it easier to find our way to a local business.

The next time I’m on a road trip with my little one and I see that he’s starting to get hungry, I’ll be able to hit a button on my phone and get directions to a nearby place that will probably be appropriate for him to sit down and eat at. I think that’s cool.

こんにちは、富士榮です。

先日、IDProで紹介されていた記事を見つつ以下のポストをしました。

パスキーはパスすべきなのか？ 結局パスキーはパスすべきなのか？

記事の中では同期ファブリックの安全性や互換性に関する課題が指摘され、デバイス紐付けパスキーという選択肢やUniversal Credential Exchangeによる相互運用性の実現に向けた取り組みの紹介をしました。

結果、エンタープライズでの利用の場合は同期パスキーよりもデバイス紐付けと管理などが現実的なのかな？？などのコメントを各所で頂いたりしました。

そんな中、RSAが最近こんなブログを公開しました。

Are FIDO Passkeys Ready for Enterprise Use?

https://www.rsa.com/ja/resources/blog/passwordless/are-fido-passkeys-ready-for-enterprise-use/

RSA社のブログ

ベンダのブログなので若干バイアスがかかる可能性は否定しませんが、まさにエンタープライズでパスキーを使えるのかどうか気になっている人は必見だと思います。

ということで簡単に見ていきましょう。

Device-Bound Passkeys vs. Synced Passkeys

まずはデバイス紐付きのパスキーと同期パスキーについて解説されています。この辺りはご存知の方が殆どだと思いますがおさらいしておきましょう。

Device-bound passkeys are generally hosted on specific ‘security key’ devices. On a device-bound passkey, key pairs are generated and stored on a single device; moreover, the key material itself never leaves that device.

With synced passkeys, the key material is saved via a so-called remote sync fabric, and the key material can then be restored on any other devices owned by the same user. The current major sync fabrics are Microsoft, Google and Apple. This means that if you were to register your Android phone as a passkey, then the corresponding key material would be available on all your other Android devices shortly after.

Synced passkeys are—in addition to the support of widely used services such as WhatsApp or Facebook—a main reason for the sharp increase in the general use of passkeys. It’s easy to see why: one user with a lot of accounts and a lot of devices can use the same synced passkey between all of them.

デバイスに紐づけられたパスキーは、通常、特定の「セキュリティキー」デバイスにホストされています。デバイスに紐づけられたパスキーでは、キーペアが生成され、単一のデバイスに保存されます。さらに、キー素材自体はデバイスから決して離れることはありません。

同期パスキーでは、鍵情報はリモート同期ファブリックと呼ばれる仕組みを介して保存され、同じユーザーが所有する他のデバイスにも復元することができます。現在、主な同期ファブリックとしては、Microsoft、Google、Appleが挙げられます。つまり、Androidフォンをパスキーとして登録すると、その後すぐに、他のすべてのAndroidデバイスでも対応する鍵情報を利用できるようになるということです。

同期されたパスキーは、WhatsAppやFacebookなどの広く使用されているサービスのサポートに加えて、パスキーの一般的な使用が急激に増加している主な理由です。その理由は簡単です。多くのアカウントとデバイスを持つユーザーは、それらすべてに同じ同期されたパスキーを使用することができます。

 

ユーザが直接サインアップして使うコンシューマ向けサービスではユーザビリティの向上を目標として同期パスキーが使われることで急激にパスキー自体の普及率が上がっている、ということですね。

The Benefits of Passkeys

パスキーの利点についてもおさらいされています。

Passkeys are an excellent MFA method: safe, fast, convenient, and users are already familiar with them. But the passkey benefit that’s beginning to get a lot of attention is that they’re phishing-resistant.

Passkeys can help organizations stop traditional phishing in its tracks: if there’s no password being used, then there’s no password to steal. And while that’s true for other passwordless MFA methods, passkeys have an added level of security provided by that synched key/service domain match.

In the U.S., phishing resistance is a major driver for government agencies. Executive Order 14028 requires phishing-resistant, passwordless authentication to secure critical infrastructure.

パスキーは優れた多要素認証の方法です。安全で、高速で、便利であり、ユーザーもすでに慣れ親しんでいます。しかし、現在注目を集め始めているパスキーの利点は、フィッシング対策に有効であるという点です。

パスキーは、従来のフィッシングを阻止するのに役立ちます。パスワードが使用されていなければ、盗むべきパスワードも存在しないからです。これはパスワード不要の他の多要素認証方法にも当てはまりますが、パスキーには、同期されたキー/サービスドメインの一致による追加のセキュリティレベルが備わっています。

米国では、フィッシング対策は政府機関にとって重要な推進要因となっています。大統領令14028では、重要なインフラストラクチャを保護するために、フィッシング対策のパスワード不要の認証が義務付けられています。

パスキーの一番の利点としてフィッシング耐性があることに触れられています。米国では大統領令でもフィッシングについての記載があるんですね。

The Challenges with Passkeys

ここからパスキーの課題について触れられています。

While passkeys provide significant advantages, they also come with a few significant challenges and problems.

For a solution that has grown up in the consumer environment, user guidance and the user experience can sometimes be a challenge.

Dialogs that ask the user to insert the passkey into the USB port or enter the PIN, for example, look different depending on the operating system and browser. Those prompts will likely make it more difficult to train users and minimize support calls.

Why not just change the prompts you ask? Because third-party service providers like RSA can’t: those prompts are set by the browser or OS vendor themselves in their own vendor (for instance, Apple sets their prompt for iOS, Google for Chrome, etc). There’s some good reason for this:  if vendors could change the prompt, then so could attackers, using an updated form to spy on users. Keeping those log-in prompts locked is an important security measure, but it can make for a one-size-fits-all approach.

The high level of phishing resistance is a clear advantage, but it can also be a distraction. Anyone who thinks that the use of passkeys suddenly makes them immune to social engineering attacks is very much mistaken. Passkeys help against one type of social engineering attack: phishing. Unfortunately, there are other variants. The attacks on MGM Resorts or Caesars Palace in Las Vegas had a social engineering component: exploiting the help desk to allow the attacker to register an MFA authenticator himself.

Attackers adapt as a matter of course. The proliferation of MFA has made phishing much less attractive, so it’s only natural that vulnerabilities around the MFA system are exploited. Such as the way users register. Anyone who thinks passkeys solve these problems is very wrong.

パスキーには大きな利点がある一方で、いくつかの重大な課題や問題も伴います。

消費者環境で成長してきたソリューションでは、ユーザーへのガイダンスやユーザーエクスペリエンスが課題となることがあります。

たとえば、パスキーをUSBポートに挿入するようユーザーに求めるダイアログやPINの入力などは、オペレーティングシステムやブラウザによって表示が異なります。このようなプロンプトは、ユーザーのトレーニングを難しくし、サポートコールを最小限に抑えることが難しくなるでしょう。

では、求めるプロンプトを変更すればよいのでしょうか?RSAのようなサードパーティのサービスプロバイダーにはできないからです。これらのプロンプトは、ブラウザやOSのベンダー自身が、それぞれのベンダーで設定しています(例えば、AppleはiOS、GoogleはChromeなど)。これにはいくつかの理由があります。もしベンダーがプロンプトを変更できるのであれば、攻撃者も同様に変更でき、最新のフォームを使用してユーザーをスパイすることが可能になります。ログインプロンプトをロックすることは重要なセキュリティ対策ですが、画一的なアプローチになりかねません。

フィッシングに対する高い耐性は明らかな利点ですが、かえって邪魔になることもあります。パスキーの使用によって、ソーシャルエンジニアリング攻撃に対する耐性が突然得られると考える人は、大きな誤解をしています。パスキーは、ソーシャルエンジニアリング攻撃の一種であるフィッシングに対する防御策となります。残念ながら、それ以外にもさまざまな種類があります。ラスベガスのMGMリゾートやシーザーズパレスに対する攻撃にはソーシャルエンジニアリングの要素がありました。ヘルプデスクを悪用して、攻撃者が多要素認証の認証装置を自分で登録できるようにしたのです。

攻撃者は当然のように適応していきます。多要素認証の普及により、フィッシングはあまり魅力的ではなくなりました。そのため、多要素認証システム周辺の脆弱性が悪用されるのは当然です。例えば、ユーザーによる登録方法などです。パスキーがこれらの問題を解決すると考える人は、大きな誤りを犯しています。

1つはUI（プロンプト）がプラットフォームによって異なる（かつサードパーティのソフトウェアでは変更ができない）ことが挙げられています。OSベンダがプロンプトを変更させないのはセキュリティの観点から見ると合理的ではありますが、OSごとにプロンプトが異なるのはユーザから見ると混乱を招く一因となっていると思います。

また、フィッシング耐性がある、というキーワードがかえって誤解を与えてしまう懸念についても記載されています。そもそもフィッシングはソーシャルエンジニアリングの一部であることを理解しないと、ソーシャルエンジニアリング全般に対する対策となっている、という誤解を招く、と指摘されています。

同じく、システムへの多要素認証手段の登録機能自体がパスキーでセキュアになる、という誤解もあるようですね。

Sync Fabrics and Cybersecurity Vulnerabilities

これも以前紹介した記事にも記載があった、同期ファブリックのセキュリティの話です。

They say that when you have a hammer, everything can look like a nail. Turing a solution—even a great solution—that was originally intended for consumer use into an enterprise application can introduce significant risk.

While reading this article, you may have had a queasy feeling at the mention of ‘sync fabric’. Your gut was right.

The fact that passkeys appear as if by magic on all devices on which the user is logged in via Apple or Google is a major red flag in the corporate environment and should raise some significant questions:

Should users be allowed to use several (possibly also privately used) devices for authentication at all? If so… How many?

Synced passkeys make restoring a “lost” passkey possible with the account recovery processes of e.g. Google or Apple. That’s great… but are these processes secure enough for you?

The Apple feature that allows users to share Passkey with friends or family is quite nice… but does this also apply to Passkeys that are used to log in to enterprise applications?

When using synced passkeys, the security of your company suddenly depends largely on the technical and organizational security of Apple and Google. Sure, there is a certain dependency anyway due to the use of iOS and Android—but synced passkeys increase this dependency considerably.

This isn’t a theoretical vulnerability, either. Last year Retool discussed how threat actors had used it to gain access to its systems: Retool wrote that the functionality means that “if your Google account is compromised, so now are your MFA codes.”

ハンマーを持っていれば、すべてが釘に見えるという。もともとコンシューマー向けに意図されたソリューション、たとえそれが優れたソリューションであったとしても、それをエンタープライズアプリケーションに転用することは、重大なリスクを伴う可能性がある。

この記事をお読みになっている方の中には、「同期ファブリック」という言葉に不安を覚えた方もいらっしゃるかもしれません。その直感は正しいです。

パスキーが、Apple や Google 経由でログインしたすべてのデバイス上に魔法のように現れるという事実は、企業環境においては重大な問題であり、次のような重要な疑問が生じます。

ユーザーは、認証のために複数の(場合によっては個人用も含む)デバイスを使用することが許されるべきでしょうか? もし許されるのであれば、何台まででしょうか?

同期されたパスキーは、Google や Apple のアカウント復旧プロセスによって「紛失した」パスキーの復元を可能にします。素晴らしいことですが、これらのプロセスは十分に安全でしょうか?

友人や家族とパスキーを共有できるAppleの機能は素晴らしいですが、これは企業アプリケーションへのログインに使用するパスキーにも適用されるのでしょうか?

同期されたパスキーを使用する場合、企業のセキュリティは突如としてAppleとGoogleの技術的および組織的なセキュリティに大きく依存することになります。もちろん、iOSやAndroidを使用しているため、ある程度の依存関係は存在しますが、同期されたパスキーは、この依存関係を大幅に高めます。

これは理論上の脆弱性でもありません。昨年、Retoolは、脅威の主体がこの機能を利用してシステムにアクセスした方法について論じています。Retoolは、この機能により「Googleアカウントが侵害された場合、MFAコードも侵害される」と書いています。

想像通り、ですがAppleやGoogleがホストする同期ファブリックを介して個人の所有物を含むデバイス間でクレデンシャルが同期されていたり、家族とクレデンシャルが共有されている環境下で企業アプリケーションを利用するのは管理者から見ると悪夢となりそうです。簡単に言うとコントロールをAppleやGoogleに握られている状態（依存している状態）は企業にとって必ずしも良い状態ではない、ということですね。

Are Passkeys Ready for Enterprise Use or Not?

結局パスキーはエンタープライズで使える状態なのか？という話です。

Whether Passkeys should be used in the company cannot be answered in a general way. Every organization is different and must balance its unique security and operational priorities.

Moreover, whether to use passkeys shouldn’t be a yes/no question. The introduction of passkeys or passwordless login in general should be used to fundamentally review an organization’s entire MFA processes. What has been good for hardware OTP tokens for 15 years is probably no longer entirely true for passkeys or other MFA methods today.

RSA believes that passkeys can be deployed for enterprise use if they align with organizational strategy and if organizations think through their answers to the following questions. We’ve seen organizations use passkeys successfully using RSA® ID Plus, our comprehensive identity and access management (IAM) platform that provides a range of passwordless options.

Because we’re a security-first organization and use Secure by Design / Secure by Default principles, we prevent using synced passkeys by default. Only device-bound passkeys are available by default in RSA environments to provide the maximum level of security out-out-the-box, and without any extra work required by admins.

パスキーを企業で使用すべきかどうかは、一概に答えられるものではありません。 組織はそれぞれ異なり、独自のセキュリティと運用上の優先事項のバランスを取る必要があります。

さらに、パスキーを使用すべきかどうかは、イエス・ノーで答えられる質問ではありません。 パスキーやパスワード不要のログインを導入することは、組織のMFAプロセス全体を根本的に見直すために使用すべきです。15年間ハードウェアOTPトークンにとって有効であったことが、パスキーやその他の多要素認証方法ではもはや完全に当てはまらない可能性が高いのです。

RSAは、パスキーを企業で使用するには、組織の戦略と一致し、組織が以下の質問に対する答えを十分に検討することが必要だと考えています。RSAは、パスワード不要のさまざまなオプションを提供する包括的なアイデンティティおよびアクセス管理(IAM)プラットフォームであるRSA® ID Plusを使用して、パスキーを成功裏に使用している組織をいくつも見てきました。

当社はセキュリティを最優先する企業であり、「Secure by Design / Secure by Default」の原則を採用しているため、同期型パスキーのデフォルトでの使用は防止しています。RSA環境では、デフォルトではデバイスに紐づいたパスキーのみが利用可能であり、管理者による追加作業を必要とせずに、最大限のセキュリティをすぐに利用できます。

 まぁ、この辺りはポジショントークが入ってきていますが、Yes/No問題ではない、という点は真理だと思います。結局のところパスキーの登場をきっかけに企業における認証戦略を見直すことが重要ってことです。

Questions Organizations Must Ask Before Using Passkeys

まぁ、最後は当然のことながら自社サービスへの誘導ですね。とは言え、企業が自問すべきことについて触れられているのでここはみなさんちゃんと考えてみましょう。

When assessing whether to introduce passkeys, organizations should ask: How are our authenticators registered? Are there processes that safely handle the ‘I lost my authenticator’ scenario? What about the classification of users, applications and data?

Passkeys are one MFA method among many.  Yes, their phishing resistance is fantastic, but can users log in with it on their remote desktops?

For these reasons and many others, it’s important that your MFA system isn’t just technically up to date, but that it also supports a wide variety of MFA methods, such as QR codes, biometrics, OTP, push messages and FIDO passkeys.

It is also important that the processes around MFA are adapted to new threats. This goes far beyond the actual MFA system: Is your help desk also safe from social engineering attacks?

If passkeys make sense to you, then we want to help. Contact us to learn more or start a free, 45-day trial of ID Plus.

 パスキーを導入すべきかどうかを評価する際、組織は次のような質問を自問すべきです。「当社の認証情報はどのように登録されているか?」「認証情報を紛失した」というシナリオを安全に処理するプロセスはあるか?」「ユーザー、アプリケーション、データの分類についてはどうなっているか?」

パスキーは、数ある多要素認証(MFA)の方法のひとつです。 確かにフィッシング対策としては素晴らしいですが、ユーザーはリモートデスクトップ上でこれを使ってログインできるのでしょうか?

こうした理由やその他の理由から、MFAシステムは単に技術的に最新であるだけでなく、QRコード、生体認証、OTP、プッシュ通知、FIDOパスキーなど、多様なMFA方式をサポートしていることが重要です。

また、MFAに関連するプロセスが新たな脅威に対応していることも重要です。これは実際のMFAシステムをはるかに超えるものです。ヘルプデスクもソーシャルエンジニアリング攻撃から安全であるか?

パスキーが適切だとお考えであれば、ぜひご相談ください。詳細についてはお問い合わせいただくか、ID Plusの45日間無料トライアルをお試しください。

確かにパスキーは多要素認証の方法の一つ、として捉えて認証戦略について考えることは大切ですね。

ということでベンダのブログではありますが、結構書いてあることは重要だと思います。

日本企業・組織においてもパスキーを前向きに捉えて導入するためにもしっかりとシナリオを設計して導入を進めてほしいところです。

 

 

“Users of the WordPress API may enjoy the abstraction — and standardization — that a SQL interface provides. If you need to query multiple WordPress sites, Steampipe’s connection aggregator will be really handy. And if you want to integrate data from WordPress with data from other APIs wrapped by other plugins in the Steampipe hub, performing literal SQL JOINs across disparate APIs is a heady experience.” — Building a Steampipe Plugin — and Powerpipe Dashboards — for WordPress

Part of the LLM series at The New Stack.

Q: I’ve joined a new company and a few months in, it’s clear my team would be better off if we adopted some best practices which are currently missing. But how can I introduce these without treading on too many toes?

If you’ve changed companies, you’ve probably found yourself in a position of observing gaps that your new team has, and how using one or two ‘best practices’ would benefit everyone. These could be to do with matters like planning or development, testing approaches, rollout strategies and more. In this issue, we cover this area in depth:

“Best practices” vs “practices”. Making the case that “best” is less important for these practices.

Common software engineering practices. For planning, development, testing and maintenance.

Two questions before adopting an engineering practice. What is the problem to be solved, what would the effect be?

Adopting a practice at team level. Most common approaches.

Risk of copy/pasting practices. There’s good reason to be skeptical about practices just because they worked well at other places.

Rejecting and dropping practices. Just because you introduced a practice doesn’t mean it should be permanently adopted.

Encouraging engineers to consider practices, as a manager. Empowering software engineers to bring practices that help the team work better.

We previously touched on related topics in these articles:

Engineering planning – an overview of RFCs, Design Documents and ADRs

Consolidating technologies – a roundup of approaches

Shipping to production – and overview of how to ship code to prod in reliable way

Migrations done well – a deepdive into this unusually challenging topic

Incident review and postmortem best practices – including a peek into the practices of the future

Healthy oncall practicess – a roundup of these, with upsides and downsides

1. Best practices vs “practices”

As a software engineer, you’ve undoubtedly heard the term “best practice.” It’s an approach or process that’s understood to help build software that’s superior in terms of speed of execution, shipping with higher quality, or building more maintainable code.

However, workplaces are different enough that what’s considered as “best practice” at one place, might fail at another.

Take a conventional best practice of doing code reviews before merging a pull request to the codebase. Code reviews have lots of upsides: catching errors, spreading knowledge, and getting feedback sooner from peers. But the biggest downside is it delays the development process by pausing it for the code review. And some tooling needs to be put in place for a review. Still, at most places the downsides aren’t big compared to the upsides.

But sometimes code reviews don’t make sense, for example in a one-person project when a code review is impractical. Likewise, when building a throwaway prototype there’s no point in a review. And there are teams with experienced software engineers for which a different approach works much better: post-commit reviews, where code reviews are done after the code is merged. This is because teams which work well together tend to get more value from not breaking the workflow of other engineers, and the number of issues which code reviews detect is low enough to justify not adding a delay to merging code.

If “best practice” code reviews are not necessarily always the optimal approach, then what about other “best practices” which are also contextual? Instead of “best practices”, I call them ‘software engineering practices.’ There are tried-and-tested software engineering practices that help with one or more dimensions of the software engineering process.

2. Common software engineering practices Several of the practices we will touch on

Below is a selection of practices, some suggested by Simon Willison, co-creator of Django. He wrote the excellent post Software engineering practices in a response to a Twitter conversation I started.

Planning

Written planning process. Before starting to code non-trivial projects, a plan is written and circulated for feedback. This plan might be called a Request for Comment (RFC,) an Engineering Requirements Document (EDD,) a Design Doc, or Architectural Decision Record (ADR.) We covered more on these in the issue Engineering planning with RFCs, Design Documents and ADRs

Standardized approach to architectural diagrams. Agreeing on the tooling or approach for creating architectural diagrams, including which symbols and approaches are not understood by some members of the group, or new joiners.

Development

Automated setting up of a new development environment. When starting at a new company or moving to a new project, it often takes painfully long to get set up in the development environment. Installing all the tools, setting the environment variables, and doing things like initializing a local database, all take time. Cloud-based developments like GitPod turn this process into a simple click. We covered GitPod’s product roadmap here. Simon Will writes: “I’ve seen developers lose hours a week to issues with the development environment. Eliminating that across a large team is the equivalent of hiring several new full-time engineers!”

Prototyping. Iterating quickly by building throwaway prototypes to validate ideas and get quick feedback from customers.

Code reviews. Before committing code, other engineers review and sign it off.

Automated code formatting. Engineers often like to format their code in different ways, and a difference in how many spaces to use for indentation, or using tabs instead, can turn into an argument and reduce the readability of the code. You can use automated code formatting solutions at the IDE-level, or solve the issue with linting.

Linting. This is the automated checking of code for stylistic and other errors which can be checked with rules. Putting linting in place before creating a pull request can help ensure the code is formatted the same way. You can also add more advanced rules such as codifying some architecture rules, for example specifying that a View class should not directly reference a Model class.

Static code analysis. Run more advanced code checks before committing, such as searching for security vulnerabilities or detecting features to be deprecated.

Templates for new projects or components. Provide ready-to-clone repository setups with the expected folder structure, and a README in place. Do the same for commonly-used components. 

Code generation for common components. When following an opinionated architecture during development, speed up code writing with tooling which generates the boilerplate code to wire things up. For example, this is an approach we took at Uber in using a code generator to output modules for our RIBs native mobile architecture.

Automated preview environments. Simon Willison writes: “Reviewing a pull request is a lot easier if you can actually try out the changes. The best way to do this is with automated preview environments, directly linked to from the pull request itself. Vercel, Netlify, Render and Heroku all have features that can do this.”

Post-commit code reviews. Code reviews are done only after the commits are made. This approach typically increases iteration speed, while still having some code reviews in place. As a tradeoff, more regressions might reach production. This practice usually works best with very small or highly experienced teams.

Cross-platform development approaches. When building a product for several platforms – say, the web, iOS, Android and desktop – an alternative is to use a framework which allows for sharing most of the code across all platforms. This could be an off-the-shelf framework, although there’s always the temptation to create something that perfectly fits your use case.

Continuous integration (CI) and continuous deployment (CD). Automatically running tests on all code changes (CI) and running the full test suite and deploying changes when committing code (CD).

Project management approaches. Using approaches like Kanban, Scrum and other frameworks like SaFE (Scaled Agile Framework,) LeSS (Large-Scale Scrum,) or your own approach. We covered more on this in How Big Tech runs tech projects and Software engineers leading projects.

Testing

Automated testing. Writing unit, integration, end to end, performance, load, or other types of tests. This approach usually increases quality and maintainability and often results in less time overall to ship software, thanks to regressions being flagged quickly.

Test-driven development. A subset of automated testing where tests are written before the actual code is.

Testing environments. Shipping code to intermediary environments for more testing, Instead of shipping straight to production. The upside is increased confidence that the code is correct. The downsides are it takes longer to ship things to production and there’s extra work maintaining the testing environments.

Testing in production, and doing it safely. Instead of using testing environments, shipping to production and using safe methods for tests, you could utilize processes like tenancies, feature flags or staged rollouts.

Testing with production data. It can be useful to be able to test with production data, without affecting production. Some companies build more sophisticated systems that can “fork” production traffic to a developer machine, and the developer can execute.

Load testing. Simulating high load for backend systems, to prepare for situations where production loads spikes and to test which systems might get overloaded or break.

Performance benchmarking and testing. For applications where performance is important, benchmark things like resource usage, i.e., CPU load, responsiveness and latency. This is important for products of which high performance is expected. An additional practice is automating performance testing and performance benchmarking to catch performance regressions right at the code changes which introduce them.

Generating test data. It can be useful to have ways of bulk-generating test data that mirrors production usage. Simon Willison has written more about this in Mechanisms for creating test data. 

We cover more on testing in How Big Tech does QA and in QA approaches across the tech industry.

Shipping

Feature flags and experimentation. Control the rollout of a change by hiding it behind a feature flag in the code. This feature flag can then be enabled for a subset of users, and those users execute the new version of the code. This approach can also be used to run A/B tests to ensure that the feature performs in-line with expectations (or that it does not cause measurable regressions in.)

Staged rollouts, also referred to as ‘progressive delivery.’ Instead of releasing new features to customers all at once, release them in stages and gather feedback. Another flavor of this is ‘blue-green deploys’ that Stripe also uses and we covered in detail.

Monitoring and alerting. Tracking a system’s health signals and alerting oncall engineers if something unusual happens.

Logging. Recording information on the system to help with debugging later.

We covered more about these topics in the deepdive Shipping to production.

Maintenance

Debugging production. Having ways to inspect the system as it runs, or to play back production use-cases, and also methods for inspecting variables and code paths to discover what the issue could be.

Documentation. Deciding what to document and how, tooling, whether documentation will be kept up to date and how to do this.

Runbook templates. For teams which are oncall, runbooks that can help the oncall engineer resolve a specific type of alert, are very helpful. Having runbook examples and templates available helps to keep these runbooks consistent and easy to navigate.

A migration runbook. Migrations are a different beast and hard to do well, which is especially true of data migrations. We cover this in depth in Migrations Done Well – including a migration checklist – and also in Real-World Engineering Challenges: Migrations. Simon Willison wrote an excellent post about Rock-solid database migrations.

3. Two questions before adopting an engineering practice

When starting at a new workplace, it’s tempting to introduce practices that worked well in your previous role. There can be a similar urge after reading about a practice which worked somewhere else, which you think would greatly benefit your new team.

However, before deciding to make a case for adopting a practice, I suggest taking a step back and asking two questions:

Question #1: What is the problem to be solved?

A simple question, but one I’ve observed engineers often neglect before starting to put a new practice in place. Be clear what the current issues are.

For engineering practices, problems worth solving can include:

Developers spend too much time doing X. Can we speed this up? X could mean waiting on scripts or code reviews, setting up development environments, writing tests, or more.

The quality of our output isn’t good and needs improving. There might be too many bugs, too many outage incidents.

The speed of execution is slow. Getting simple things done could take longer than necessary due to continuous processes, tools, or workarounds.

Developers context switch too much. Context switching for the wrong reasons. 

It’s hard to do X. It could be hard and time-consuming to debug a part of the system, to resolve certain types of outages, or to get people on the same page. It’s ideal if you can verbalize the consequences of things being hard to do.

Once you’ve verbalized the problem, what is its impact? If it was solved or made less of an issue, what would improve? Things that teams – and companies – tend to care about include:

Revenue generation

Business goals

Regressions

Reliability

Outages

Iteration speed

And how pressing is the issue? For example, if it’s hard to debug one of the services, but that service is barely touched and only needs debugging once every few months, then perhaps it’s not urgent to solve it.

Question #2: If adopted, what would be the effect of a practice? 

When there’s a problem to solve, wait before moving ahead with the engineering practice that helps with it. That’s because every new practice introduced will have an impact on how people and teams work.

For example, say your team has no code reviews in place and also there are quality issues, like too many bugs being shipped. By putting mandatory, pre-merge code reviews in place, those quality issues could be reduced. However, now there will be new issues, such as people having to wait on code reviews, figuring out what to do when no reviewers are available, or what to do when there’s a lot of debate about a code review which stalls the process.

There are a few common ways to find out what may happen if a practice is put in place:

Do a “pre-mortem.” Make hypotheses about how things can change. You know your team, your company, and the people. Imagine you put this practice in place: what issues could it cause? For example, you might know there’s a team member who’s outspoken about a practice you want to bring in, which could be a source of conflict if not done right. If your team has less experienced engineers and you are proposing a practice which will make the development workflow more complex – like suggesting a certain architecture pattern to follow – then they’ll need more hand-holding and may struggle to follow the approach without support.

Consult case studies of the new practice being introduced elsewhere. There are plenty of teams and companies which are open to sharing their experiences with practices and how they worked. Check out engineering blogs, conference talks and recordings to gather details on how things went at other companies. However, be aware that engineers often focus more on success stories and less on what went wrong when acting as company representatives.

Talk with people who already use this practice. The problem with case studies is they tend to give an overly rosy picture of a practice. To get the full picture, it helps to talk to engineers who have used the practice, ideally in a similar environment to yours. If you can find colleagues who adopted the practice, also talk with them. If you can’t access people internally, don’t be afraid to reach out externally to engineers who have written about using this practice. For example, when I was at Skyscanner, a few engineers reached out to Spotify to learn more about how engineering teams organized there. A few Skyscanner engineers even traveled to Stockholm to meet and discuss it with Spotify engineers, who were very helpful. And it all started with just a cold email to an engineer!

4. Adopting a practice at team level

Read more

こんにちは、富士榮です。

大学ICT推進協議会（AXIES・アクシーズ）の参加申し込みなどが始まっていますね。

https://axies.secretari.jp/conf2024/

いやぁ、奈良すぎる。ということで今年は奈良ですね。（去年は名古屋だった）

アイデンティティとの絡みで言うと、AXIESの中の認証基盤部会が毎年学術機関におけるデジタルアイデンティティについてのトレンドやあり方について発表しています。

私も毎年現地に行っていますので今年も行こうと思います。みなさんぜひお会いしましょう。 （12月ですが）

こんにちは、富士榮です。
パスキーのテストをする際によく使うpasskeys.devに、端末（プラットフォーム・ブラウザ）がどんな機能をサポートしているのかを確認するためのページができています。
WebAuthn Features and Capability Detection https://featuredetect.passkeys.dev/
これを使うとブラウザが何をサポートしているのかが一目でわかります。 確認できるのは、以下の通りです。 Features Get Client Capabilities Conditional Get (Autofill UI) Conditional Create (Opportunistic Upgrades) Related Origin Requests toJSON().Method Parse JSON Request Options Parse JSON Creation Options Client Capabilities Passkey Platform Authenticator User Verifying Platform Authenticator Hybrid Transports Extension: PRF
試しに手元のデバイス・ブラウザで確認してみました。
MacOS 14.6.1 / Safari MacOS 14.6.1 / Firefox 129.0.1


Android 14 / Chromeブラウザ
iOS 17.5.1 / Safari


うーむ。まぁわかってはいましたが先日の「パスキーはパスすべきなのか？」に書いた通りまだまだサポートレベルにばらつきはありますよねぇ。

こんにちは、富士榮です。

引き続き選択的開示に関する調査論文を読んでいきます。
Selective disclosure in digital credentials: A review
https://www.sciencedirect.com/science/article/pii/S2405959524000614
今回は結論部分なので、これでこのシリーズは終わりです。
まとめとして以下の事項が書かれています。
Comprehensive literature review: We gave a comprehensive literature review on the broad topic of selective disclosure, identifying seminal works and future trends; 包括的な文献レビュー:選択的開示という幅広いテーマについて、包括的な文献レビューを提示し、画期的な研究や今後の動向を特定しました。 Differentiation and categorization: We presented a differentiation and categorization of different types and formats for selective disclosure through RQ1. We introduced a new category, ZKP, and showed how combining different methods can improve selective disclosure. We showed the comparative strengths and weaknesses of each selective disclosure method and gave tables to explain the performance of each selective disclosure approach; 差別化とカテゴリー化:RQ1を通じて、選択的開示のさまざまな種類と形式の差別化とカテゴリー化を提示しました。ZKPという新しいカテゴリーを導入し、異なる方法を組み合わせることで選択的開示がどのように改善されるかを示しました。各選択的開示方法の比較上の長所と短所を示し、各選択的開示アプローチのパフォーマンスを説明する表を提示しました。 選択的開示の手法として属性単位でクレデンシャルを作るアトミッククレデンシャル、署名ベースの方式、ハッシュ値ベースの方式が紹介された上で、ゼロ知識証明との組み合わせについても紹介されていましたね。 Application across formats: We illustrated through RQ2 how different methods of selective disclosure are applied across various formats. VCs and ACs are currently the most used formats. Hash-based and signature-based methods are the most commonly used approaches for selective disclosure; フォーマットをまたいだ適用:RQ2では、さまざまなフォーマットにまたがって、選択的開示の異なる方法がどのように適用されているかを説明しました。現在、最もよく使用されているフォーマットはVCとACです。選択的開示の最も一般的なアプローチは、ハッシュ値ベースと署名ベースの方法です。 クレデンシャルフォーマットによってどんな手法が取られているか、について解説がありました。ハッシュ値ベース、署名ベースでの選択的開示が多かったですね。 Necessity and benefits of ZKP: We showed how ZKP is necessary for implementing specific signatures, but that is not necessary to achieve selective disclosure through RQ3. Trends suggest that ZKP gives an added benefit to selective disclosure and can be implemented as part of the solution for selective disclosure; ZKPの必要性と利点:ZKPが特定の署名を実装するために必要であることを示しましたが、RQ3による選択的開示を実現するには必ずしも必要ではありません。傾向から、ZKPは選択的開示に追加の利点をもたらし、選択的開示のソリューションの一部として実装できることが示唆されます。 選択的開示にゼロ知識証明を利用するかどうか、について方式と合わせて分析していましたね。 Future trends in digital identity: We showed that future trends for implementing identity and credentials tend to be focused on using blockchain through RQ4, but that there are benefits and drawbacks in using it; デジタルアイデンティティの今後の動向:RQ4を通じて、アイデンティティとクレデンシャルの実装に関する今後の動向はブロックチェーンの利用に重点が置かれる傾向にあることを示したが、それを利用することには利点と欠点がある。 ブロックチェーンの利用動向について分析がされていました。2018年以降、選択的開示に関してブロックチェーンの利用が増えてきている、という分析でした。 Identification of research gaps: We identified critical gaps in current research, from technical to regulative gaps. 研究のギャップの特定:技術的なギャップから規制上のギャップまで、現在の研究における重大なギャップを特定しました。 研究と実装や規制の間にはまだまだギャップがある、という話でした。まだまだ技術の進歩が必要ですね。

そして、以下の文章で締め括られています。

Currently, this research area is expanding, and there is still room for improvement for all the defined categories of methods for selective disclosure. There is no clear winner and the “best” universal solution. We encourage researchers to improve on the existing methods, consider new methods or revisit older ones, and even consider methods that are quantum-resistant for the future. The focus should be on finding methods that satisfy all requirements for selective disclosure schemes and specific regulations.
In the future, standardizing credentials will result in interoperable solutions and improve the development of methods for achieving selective disclosure. With this paper, our goal was to create a starting point for researchers interested in achieving selective disclosure in the digital credential world.

現在、この研究分野は拡大しており、選択的開示のための定義された方法のすべてのカテゴリーにおいて、まだ改善の余地がある。明確な勝者や「最善」の普遍的なソリューションは存在しない。研究者には、既存の方法を改善し、新しい方法を検討したり、古い方法を再検討したり、さらには将来的な量子耐性のある方法も検討することを推奨する。選択的開示スキームのすべての要件および特定の規制を満たす方法を見つけることに焦点を当てるべきである。

将来的には、クレデンシャルの標準化により相互運用可能なソリューションが実現し、選択的開示を実現するための方法の開発が改善されるでしょう。本論文の目的は、デジタルクレデンシャル世界における選択的開示の実現に関心を持つ研究者の出発点となることでした。

将来に向け、継続的な技術開発が望まれますね。

どうやらディープフェイクによる経済的損害が半端ないことになってきているようです。

SBBITの記事『ディープフェイクの「精度向上」がヤバすぎる、2027年には被害額6兆円超に』によると、以下のような点が指摘されています。

記事の内容 ディープフェイクの精度向上 ディープフェイク技術はAIの進化により急速に発展。 初期のディープフェイクは品質が低く、明らかにフェイクと分かるものだった。 2018年から2019年にかけて、AIによる画像生成技術が向上。 GANの登場により画像の品質が大幅に向上。 2020年以降、Transformers技術により長時間動画の一貫性が向上。 2023年にはディープフェイクコンテンツが前年比3000％増加。 ディープフェイク悪用による被害 デロイトの推計によると、2023年の詐欺被害額は123億ドルから2027年には400億ドルに達する見込み。 年平均成長率は32％で、4年間で被害額が3倍以上に。 新しい生成AIツールにより、低コストでディープフェイクが作成可能。 特に金融サービス業界が標的となるケースが増加。 2023年にフィンテック業界でのディープフェイク事案は700％増加。 音声ディープフェイクによるコンタクトセンター詐欺の年間損失は約50億ドル。 2024年にはディープフェイク関連の事案が前年比60％増加し、全世界で15万件に達する予測。 同意なしの性的コンテンツや本人確認書類の偽造が懸念される。 ダークウェブで詐欺ソフトウェアが販売される闇産業が形成。 実際に起きたディープフェイクによる詐欺被害 企業幹部を狙ったディープフェイク詐欺が増加中。 世界最大の広告代理店グループWPPのCEOを狙ったWhatsApp詐欺の事例。 香港での企業幹部なりすまし事件で数千万ドルの被害の事例。 AIを悪用したサイバー攻撃が増加しているとの報告。 ディープフェイクだけではないAIによるサイバー攻撃 Ivantiの調査によると、企業の多くはAIを悪用したサイバー攻撃が増加していると報告。 AI駆動のサイバー攻撃は今後さらに増える見込み。 特に警戒される脅威としては、フィッシング（45％）、ソフトウェアの脆弱性を狙った攻撃（38％）、ランサム攻撃（37％）、APIの脆弱性を狙った攻撃（34％）が挙げられる。 ディープフェイク対策の現状 銀行など金融機関はAIや機械学習を用いた不正検知システムを導入。 JPモルガンはメール詐欺検出に大規模言語モデルを使用。 マスターカードは取引の正当性を予測する「Decision Intelligence」ツールを開発。 既存のリスク管理フレームワークは新たなAI技術に対応しきれない可能性がある。 国を挙げて取り組むディープフェイク対策 目視でのディープフェイク判別が困難になっているとの指摘あり。 OpenAIが自社のAIを使ったディープフェイク検出ツールを提供予定。ただし、ディープフェイクは単一ツールで作られることは稀で、このようなツールの有効性には限界がある。 C2PAイニシアチブがAI生成コンテンツの制作過程を食品成分表示のような形で示す規格を開発中。 英国政府が「ディープフェイク検出チャレンジ」を実施。 一般向けの啓発活動が進められている。 アイデンティティの観点からの感想

生成AIがアイデンティティにもたらすインパクトは多岐にわたります。ディープフェイクはその一側面です。

対ディープフェイク管理策ということでは、

発信者認証 音声や顔画像によって人間が判断するのではなく、重要なトランザクションの前には情報発信者を必ず高度認証で認証（技術的対策） これを担保するための組織的対策 身元確認書類の偽造対策としてデジタル化を推進 拡散される情報自体の性質の明示 こうした対策を実施するための人的対策

などが必要となるでしょう。

発信者認証

発信者認証の例としては、電話やビデオなどでの依頼に対して、必ずCIBA1を使って、名乗っている人の事前に登録されたデバイスにプッシュ通知を送ってユーザ認証を行うなどがあげられます。

一方、このようなことを電話口の人が要求しても、その人がクビになったりしないことを保証してあげるというようなことも重要です。このような詐欺の典型的な手口として、電話口の人が逡巡すると「会社の存亡がかかっているのだ。今すぐやらないとお前はクビだ。」などとプレッシャーをかけるのですが、こうしたプレッシャーから守って上げる必要があるのです。これは技術的対策だけでは難しく、社内規定などの組織的対策が必要になります。

また、ディープフェイクによる身元確認書類の偽造に対しては、デジタル署名をつけたものへの移行が有効です。幸いにして日本では公的個人認証や、デジタル庁デジタル認証アプリなどが使えますから、こうしたものに依拠して、高いレベルの身元確認を行っていくことが求められると思います。

拡散される情報自体の性質の明示

拡散される情報自体の性質の明示は、その情報がどのように生成されたのかと情報の発信源が誰なのかということの両方があると思います。これは、アイデンティティの一貫性を守るうえでとても重要です。たとえば、不同意の性的なコンテンツや、犯罪を犯している場面の動画を作って拡散されたらどうなるでしょう？それが信じられてしまえば、その人物への他者の認識が変わってしまい、信頼が失墜するのは間違いありません。

こうしたことを敷設役割を担うのが、C2PAやOriginator Profile です。その動画や画像が生成AIによるものであるとか、発信者が誰であるかなどを示してくれるからです。ただ、このあたりは言論の自由とのからみでは少し注意が必要です。

C2PAとOriginator Profile（OP）は、デジタルコンテンツの信頼性を向上させる技術ですが、言論の自由に対してそれぞれ異なる影響を持つ可能性があります。

C2PAと言論の自由

C2PAは、デジタルコンテンツの出所や編集履歴を証明するための技術であり、フェイクニュースやディープフェイクの拡散を防ぐことを目的としています。しかし、この技術が誤用されると、言論の自由に対する制約を引き起こす可能性があります。例えば、C2PAのシステムがジャーナリストの身元を特定するために使用され、政府がそれを利用して言論を制限することが懸念されています。また、C2PAによるコンテンツの追跡が、特定の法律を強制するために利用される可能性もあります。

Originator Profileと言論の自由

Originator Profileは、ウェブコンテンツの発信者の真正性と信頼性を確認するための技術です。これは、偽情報や広告詐欺の抑止を目的としていますが、発信者の身元を特定することで、匿名性が失われ、言論の自由が制約される可能性があります。特に、発信者の情報が不適切に使用されると、自己検閲を促すことになりかねません。

言論の自由への影響 プライバシーの懸念: どちらの技術も、発信者の情報を収集・管理するため、プライバシーの侵害が懸念されます。これにより、発信者が自由に意見を表明することが難しくなる可能性があります。 誤用のリスク: 技術が政府や他の権力者によって誤用されると、言論の自由が制限されるリスクがあります。特に、ジャーナリストや活動家がターゲットにされる可能性があります。 技術の透明性と説明責任: これらの技術がどのように使用され、データがどのように管理されるかについての透明性が求められます。適切な説明責任がない場合、言論の自由が脅かされる可能性があります。

これらの技術は、デジタルコンテンツの信頼性を高めるために重要ですが、言論の自由を守るためには、その使用方法や管理について慎重な配慮が必要です。

人的対策

最後の人的対策もとても重要です。せっかく技術的対策を行っても、それが使われなければ意味がありませんから。ただ、これはなかなか難しいです。社員など組織内の構成員に対しては、組織的教育や罰則などで強制していくことも可能ではありますが、一般大衆に対してはなかなか難しい。このあたりは課題かなと思います。

おわりに

攻撃側のツールの能力は指数的に進化する一方、人間のスキルはそのように進化しないので、技術の支援なくスキルのみによる対抗には無理があります。なので、技術的対策を強力に推進していくことが必要です。

その一方、社会的発信に関しては言論の自由との関係も重要です。なので、やりすぎは禁物です。また、人的対策の困難性も意識しておく必要があります。

こうしたことを総合的に鑑みて、バランスよく対策を施していくことが肝要です。

こんにちは、富士榮です。

Entra Verified IDのFace check機能の正式リリースについて先月書きましたが、アナウンス的にはEntra Suiteが正式リリースでFace check自体はPublic Previewの状態だったみたいです。

https://idmlab.eidentity.jp/2024/07/entra-verified-id.html

ということで、（こんどこそ）正式リリースになったようです。

https://techcommunity.microsoft.com/t5/microsoft-entra-blog/face-check-is-now-generally-available/ba-p/4175880

MicrosoftのBlogより

ということでHappy face check lifeを！

こんにちは、富士榮です。

引き続き選択的開示に関する調査論文を読んでいきます。

Selective disclosure in digital credentials: A review

https://www.sciencedirect.com/science/article/pii/S2405959524000614

今回は研究と実装の間に、、、という話です。

論文中には研究の対象となっている選択的開示技術と社会実装における各種規制などとのギャップについて記載があります。

例えばGDPRとCRPA（記載時点ではCCPA）などを鑑みると選択的情報開示の実装には以下の要件が求められると書かれています。

Disclosing attributes from at least two separate credentials issued by the same or different issuers; Proving disclosed attributes belong to the subject presenting them; Ensuring disclosed attributes are unlinkable from multiple presentation sessions; Proving that disclosed attributes belong to the appropriate credential. 同一または異なる発行者によって発行された少なくとも2つの別個のクレデンシャルから属性を開示すること。 開示された属性が提示した対象に属することを証明すること。 開示された属性が複数の提示セッションからリンクできないことを保証すること。 開示された属性が適切なクレデンシャルに属することを証明すること。 本論文で分析した対象の中には要件を達成しているものもあるが全く考慮されていないものも存在している、ということです。 要するに要件が整理されてくるより前に要素技術の研究が始まって自由に仕様を作っていっていた、ということなんでしょう。。
同じく実装を見ると以下のようなギャップに関しても指摘されています。 Balancing privacy with transparency — one of the biggest challenges is balancing the need for privacy with the requirements for transparency; Regulatory compliance — as laws evolve, ensuring that technology complies with international, federal, and national regulations is becoming increasingly complex; Security risks — implementing selective disclosure increases the complexity of the encryption system, potentially introducing new vulnerabilities; Scalability and efficiency — certain solutions may need to be more scalable and efficient for widespread use. There is a need for a more robust system that can handle large volumes of data. プライバシーと透明性のバランス — 最大の課題のひとつは、プライバシーの必要性と透明性の要件のバランスを取ることです。 規制への準拠 — 法規制の進化に伴い、テクノロジーが国際的、連邦、および国家レベルの規制に準拠することはますます複雑化しています。 セキュリティリスク — 選択的開示を実施すると、暗号化システムの複雑性が増し、新たな脆弱性が潜在的に生じる可能性があります。 拡張性と効率性 — 特定のソリューションは、より広範な用途に利用できるよう、より拡張性と効率性を高める必要があるかもしれません。 大量のデータを処理できる、より強固なシステムが必要です。
確かに3つ目の実装の複雑性がますことでセキュリティ・リスクが高まる、という視点はあるなぁ、、と思ってみていました。 SAMLの脆弱性の話など、どうしても頑張って実装する必要がある仕様って、仕様は正しいんだけど実装で間違える、ってことが起きがちですね。
また、PQCの話も重要な点として指摘されています。 署名ベースの場合は、アルゴリズムの強度を考えないとポスト量子セキュアとは言えませんね。

いよいよ次は結論というかまとめの部分です。

Twelfth in the News Commons series

Last week at DWeb Camp, I gave a talk titled The Future, Present, and Past of News—and Why Archives Anchor It All. Here’s a frame from a phone video:

DWeb Camp is a wonderful gathering, hosted by the Internet Archive at Camp Navarro in Northern California. In this post I’ll give the same talk, adding some points I didn’t get to my 25-minute window. Here goes.

For journalism, news is bad:

Revenue sources are going away:

Wages suck:

So does employment:

But it’s looking up for bullshit and filler:

So what can we do?

There’s the easy choice—

But bear in mind that,

News only sucks as a big business.
But not as a small one.

For example, here:

Bloomington, Indiana. That’s where my wife and I are living while we serve as visiting scholars with the Ostrom Workshop at Indiana University. I’ve lived in many college towns, and this is my favorite, for many reasons. One is the quality and quantity of journalism here. And that’s on top of what IU does, which is awesome. ( I speak especially of the IU’sMedia School, the Arnolt Center for Investigative Journalism, and the Indiana Daily Student, or IDS—the oldest as well as one of the best of its breed.) I mean all the periodicals, broadcasters, podcasters, bloggers, and civic institutions that contribute to the region’s News Commons. I list participants here. (If you’re not on one of those lists, tell me and I’ll add you.)

Yet in 2017, Columbia Journalism Review produced an interactive map of America’s “news deserts” and made one of them Monroe County, most of which is Bloomington:

In fact, Bloomington did have a daily paper then: the Herald-Times. It’s still here in print and online. But, as “Breaking the News: The Past and Uncertain Future of Local Print Journalism” explained last year in a huge and well-curated exhibit at the Monroe County History Center,

the Herald-Times has shrunk quite a bit,  while some “enterprising individuals” are taking up the slack—and then some. One they single out is Dave Askins of the B Square Bulletin:

Dave’s beat is city and county government. His “almost daily” newsletter and website are free, but they are also his business, and he makes his living off of voluntary support from his readers. More importantly, Dave has some original, simple, and far-reaching ideas about how local news should work. That’s what I’m here to talk about.

We’ll start with the base format of human interest, and therefore also of journalism: stories.

Right now, as you read this, journalists are being asked the same three words, either by themselves or by their editors:

I was 23 when I got my first job at a newspaper, and quickly learned that there are just three elements to every story:

Character Problem Movement

That’s it.

The character can be a person, a ball club, a political party, whoever or whatever. They can be good or bad, few or many. It doesn’t matter, so long as they are interesting.

The problem can involve struggle, conflict, or any challenge—or collection of them—that holds your interest. This is what gets and keeps readers, viewers, and listeners engaged.

The movement needs to be toward resolution, even if it never gets there. (Soap operas never do, but the movement is always there.)

Lack any one of those three and you don’t have a story.

So let’s start with Character. Do you know who this is?

Probably not. (Nobody in the audience at my talk recognized him.)

He’s Pol Pot, who gets bronze medal for genocide, given the number of people he had killed (at 1.5 to 2 million, he comes in behind Hitler and Stalin) and a gold medal for killing the largest percentage of his own country’s population (a quarter or so).* His country was Cambodia, which he and the Khmer Rouge regime rebranded Kampuchea while all the killing was going on, from 1975 to 1979.

The first we in the West heard much about the situation was in the mid-70s, for example by this piece in the May 9, 1975 edition of The New York Times:

The link here goes to the paper’s TimesMachine archives. Seeing it requires a subscription. We’ll discuss this more below.

It’s a front-page story by Sydney Shanberg, who himself becomes a character later (as we’ll see).

The first we (or at least I) heard about the genocide was sometime in 1976 or ’77, while watching Hughes Rudd on the CBS Morning News. He said something like this:

Wierdly, it wasn’t the top story. As I recall, it came before an ad break. It was as if Rudd had said, “All these people died, back after this.”

So I went to town (Chapel Hill at the time) and bought a New York Times. There was something small on an inner page, as I recall. (I’ve dug a lot and still haven’t found it.)

But Time, the weekly news magazine, did cover at least the beginning of the genocide, in the April 19, 1976 issue. (By the way, all hail to Time for its open archives. This is a saving grace I’ll talk about later, and much appreciated.) Here is how that story begins:

Note how Prince Norodom Sihanouk stars in the opening sentence. He’s there because he was a huge character at the time. (Go read about him. He was a piece of work, but not a bad one.) Pol Pot doesn’t appear at all. And the number of dead doesn’t show up until the following paragraph:

Imagine if one day every plane in the world crashed, killing half a million people. That would be news, right? There would be a giant WTF? or HFS! in the thought balloons over all the world’s heads. But, while the systematic murder of more than half a million people could hardly be a bigger deal, it wasn’t much of a story. Lacking a Hitler or a Stalin to cast in the leading role, Time borrowed some interest in deposed characters (Sihanouk and Nol) to pull the reader down to the completely awful news. But there were no journalists reporting from Cambodia at the time, and no news about who the dead people were, or who killed them. The whole place was a black news hole.

So, lacking the required story elements, news coverage of Cambodia in the late ’70s was sparse. Pol Pot didn’t show up in the Times until this was posted in the Sunday paper on October 7, 1977, deep in what might as well have been called the Unimportant section:

Note that “Top Spot to Pol Pot” gives more space to Cambodia’s military conflicts with neighbors than to the known fact that Pot’s regime was already a world-class horror show.

The Times‘ most serious coverage of Cambodia in those years was in opinion pieces. For example, an editorial titled The Unreachable Terror in Cambodia ran on July 3, 1978, and only mentions Pol Pot in the third paragraph. Holocaust II!, by Florence Graves, ran on Page 210 of the November 26, 1978 Sunday Magazine. It begins, “They don’t talk about extermination ovens or about freaky medical experiments or about lampshades fashioned from human skin. But the Cambodian refugees do talk about forced labor camps, about “deportations,” and about mass executions.” Later she adds, “It’s a horror story which, for the most part, has gone unreported in the American press.” Because, again, it wasn’t a story.

But then this came, in January 1980:

In the opening paragraph, Sydney Schanberg writes, “This is a story of war and friendship, of the anguish of a ruined country and of one man’s will to live.” At last, we had a human being, an interesting and sympathetic character we could relate to. His problem was massive, and the resolution was heartwarming. The whole story was so compelling that four years later it became one of the great movies of its time (with seven Academy Award nominations):

And, just as Anne Frank‘s story fueled greater interest in The Holocaust, Dith Pran‘s story fueled greater interest in the Cambodian Genocide.

Lesson: in story-telling, it’s hard for a statistic to do a human’s job. Another: facts alone make lousy characters.

Now let’s go closer to home:

Trump is a genius at all that stuff. And he keeps his ball rolling by making shit up constantly. Think about it: you’re a lot more free to tell stories—and to move them along—if facts don’t matter to you. Also when your personal story transcends the absent truths in what you say. Trump’s story is a great one. Really. It is. He made himself one of the biggest (perhaps the biggest) and richest personal brands in the history of business and then got elected president by blowing through fourteen (14!) Republican primaries and caucuses and then winning the general election. As for why, here’s how I put it in Stories vs. Facts:

Here are another three words you need to know, because they pose an extreme challenge for journalism in an age when stories abound and sources are mostly tribal, meaning their stories are about their own chosen heroes, villains, and the problems that connect them: Facts don’t matter.

Daniel Kahneman says that. So does Scott Adams.

Kahneman says facts don’t matter because people’s minds are already made up about most things, and what their minds are made up about are stories. People already like, dislike, or actively don’t care about the characters involved, and they have well-formed opinions about whatever the problems are.

Adams puts it more simply: “What matters is how much we hate the person talking.” In other words, people have stories about whoever they hate. Or at least dislike. And a hero (or few) on their side.

These days we like to call stories “narratives.” Whenever you hear somebody talk about “controlling the narrative,” they’re not talking about facts. They want to shape or tell stories that may have nothing to do with facts.

But let’s say you’re a decision-maker: the lead character in a personal story about getting a job done. You’re the captain of a ship, the mayor of a town, a general on a battlefield, the detective on a case. What do you need most? Somebody’s narrative? Or facts?

The stories we tell depends on what we need.

Journalists need to fill pages and airtime.

Donald Trump needs to win an election. He needs voters and their votes. He’ll get those with stories—about himself, and about who he’d like you to hate along with him—just like he always has.

As characters go, Kamala Harris is a much tougher opponent than Joe Biden, because she’s harder for Trump to characterize, and she has plenty of character on her own.

And now both make good stories because they’re in a fight against each other.

Things are different in places like Bloomington, where people live and work in the real world.

In the real world, there are potholes, homeless encampments, storms, and other problems only people can solve—or prevent—preferably by working together. In places like that, what should journalists do, preferably together?

To clarify the options, look at journalism’s choice of sources and options for expression. For both, you’ve got facts and opinions. A typical story has a collection of facts, and some authoritative source (a professor, an author, or whatever) providing a useful opinion about those facts. Sometimes both come from the same place, such as the National Weather Service. So let’s look at different approaches to news against this background:

Here is roughly what you’ll get from serious and well-staffed news organizations such as The Wall Street Journal, The New York Times, the BBC, and NPR:

Mostly facts, but some opinions, typically on opinion pages and columns, or in support of a fact-based story.

How about research centers that publish studies and are often sourced by news organizations? Talking here about Pew, Shorenstein, Brookings, Rand. How do they sort out? Here’s a stab:

Lots of facts, plus one official opinion derived from facts. Sure, there are exceptions, but that’s a rough ratio.

How about cable news networks: CNN, Fox News, MSNBC, and their wannabes? Those look like this:

These networks are mostly made of character-driven shows. They may be fact-based to some degree, but their product is mostly opinion. Facts are filtered out through on-screen performers.

Talk radio of the political kind is almost all opinion:

Yes, facts are involved, but as Scott Adams says, facts don’t matter. What matters are partisan opinions. Their stories are about who they love and hate.

Sports talk is different. It’s chock full of facts, but with lots of opinions as well:

Blogs like the one you’re reading? Well…

I do my best to base my opinions on facts here, but readers come here for both.

Finally, there’s Dave Askins. Here’s what he gives you in the B Square Bulletin:

Dave is about facts. And that’s at the heart of his plan for making local journalism a model for every other kind of journalism that cares about being fully useful and not just telling stories all the time. One source he consulted for this plan is Bloomington Mayor Kerry Thompson. When Dave asked her what might appeal about his approach, she said this:

Dave sees facts flowing from the future to the past, like this:

The same goes for lots of other work, such as business, scholarship, and running your life. But we’re talking about local journalism here. For that, Dave sees the flow going like this:

Calendars tell journalists what’s coming up, and archives are where facts go after they’ve been plans or events, whether or not they’ve been the subjects of story-telling. That way decision-makers, whether they be journalists, city officials, or citizens, won’t have to rely on stories alone—or worse: their memory, or hearsay.

Dave has started work on both the future and the past, starting with calendars. On the B Square Bulletin, he has what’s called the Big Big Calendar. Here is how this month started:

Here are the sources for those entries:

Every outfit Dave can find in Monroe county that publishes a calendar and also has a feed is in there. They don’t need to do any more work than that. I suspect most don’t even know they syndicate their calendars automatically.

On the archive side, Dave has BloomDocs, which he explains this way on the About page:

BloomDocs is a public document repository. Anyone can upload a file. Anyone can look at the files that have been uploaded. That’s it.

What use is such a thing?

For Journalists: Journalists can upload original source files (contracts, court filings, responses to records requests, ordinances, resolutions, datasets) to BloomDocs so that they can link readers directly to the source material.

For Residents: Residents who have a public document they’d like to make available to the rest of the world can upload it to BloomDocs. It could be the government’s response to a records request. It could be a slide deck a resident has created for a presentation to the city council.

For Elected Officials: Elected officials who don’t have government website privileges and do not maintain their own websites can upload files to BloomDocs as a service to their constituents.

For Government Staff: Public servants who have a document they would like to disseminate to the public, but don’t have a handy place to post it on an official government website, or if they want a redundant place to post it, can upload the file to BloomDocs.

A future vision: “Look for it on BloomDocs” is a common answer to the question: Where can I get a copy of that document?

Dave also doesn’t see this as a solo effort. He (and we, at the Ostrom Workshop, who study such things) want this to be part of the News Commons I’ve been writing about here (this post is the 12th post in the series). In that commons, the flow would look like this:

All the publishers, radio and TV stations, city and county institutions, podcasts, and blogs I show there (and visit in We Need Wide News and We Need Whole News) should have their own arrows that go from Future to Past, and from Calendars to Archives. And when news events happen, which they do all the time and not on a schedule, those should flow into archives as well. We need to normalize archiving as much as we can.

Which brings us to money. What do we need to fund here?

Let’s start with the calendar. Dave’s big idea here is DatePress, which he details at that link. DatePress might be something WordPress would do, or somebody might do with WordPress (the base code of which is open source). I’m writing on WordPress right now. Dave publishes the B Square on WordPress. I’ll bet the websites for most of the entities above are on it too. It’s the world’s dominant CMS (content management system). See the stats here and here.

On the archiving side, BloomDocs is a place to upload and search files, of which there are hundreds so far. But to work as a complete and growing archive, BloomDocs needs its own robust CMS, also based on open source. There are a variety of choices here, but making those happen will take work, and that will require funding. Archives, being open, should also be backed up at the Internet Archive as well.

One approach is to fund development of DatePress and BloomDocs, and to expand the work Dave already leads.

Another is to drop the long-standing newspaper practice of locking up archives behind paywalls. (In Bloomington this would apply only to the Herald-Times). The new practice would be to charge for the news (if you like), but give away the olds. In other words, stop charging for access to archives. Be like Time and not like the Times and nearly every other paper. (I first brought this up here in 2007.)

Another is to look for ways readers, viewers, and listeners can pay value for value, and not confine thinking only to advertising, subscriptions, and subsidy. There are ideas out there for that, but I’ll save them for my next post in the News Commons series.

What matters for now is that all the ideas you just read about are original, and apply in our new digital age.

These ideas should also open our minds toward new horizons that have been observed insufficiently by journalism since the word first entered popular usage in the middle of the 19th century.

The challenge now isn’t to save the newspapers, magazines, and TV news reports that served us before we all started carrying glowing rectangles in our pockets and purses—and getting our news on those. It’s to make facts matter and keep mattering, after stories that use facts move off the screens, speakers, and earphones that feed our interests and appetites.

If you’re interested in weighing in or helping with this, leave a comment below or talk to me. I’m first name (well, nickname) at last name dot com.

*Based on estimates. Nobody knows for sure. Here’s Wikipedia.

So I went to the ChatGPT website to ask a question and got hit with a popover promo for the new Mac app version. So I got it. Here is the dialog that followed my first question (which is boring, so we’ll skip it), copied over from the ChatGPT website, where I went after this frustrating encounter:

Can I move my chat history from the web to here?

ChatGPT said:

Unfortunately, there’s no direct way to move chat history from one platform to another. However, if there’s specific information or conversations you’d like to continue or reference, you can copy and paste the relevant parts here, and we can pick up from there! If there’s anything specific you want to discuss or continue from a past conversation, just let me know.

You said: where here is my chat history? ChatGPT said:

Your chat history here is session-based, meaning it exists only for the duration of our current conversation. Once the session ends, the chat history isn’t saved. However, I can remember key details or preferences across sessions if you want, and you can also start fresh conversations at any time.

If there’s something specific you want to continue discussing from a previous session, you might want to copy and paste that information here.

You said: That’s useless. Thanks anyway. ChatGPT said:

I understand that can be frustrating. If you need any help or want to continue discussing something specific, I’m here to assist. Just let me know!

You said: I’m paying OpenAI for ChatGPT. I would hope that the app would offer advantages over the Web page. ChatGPT said

I can see why you’d expect some added benefits from the app version. While the app and web versions of ChatGPT share many of the same core functionalities, the app offers convenience for on-the-go use, such as notifications and a more optimized interface for mobile devices. However, both platforms currently have similar features in terms of chat history and session management.

If you have suggestions or feedback on features you’d like to see, sharing that with OpenAI could be helpful. They are continually improving the platform, and user feedback is a big part of that process.

Is there anything specific you’d like to see improved or added to the app?

You said: Yes. chat history. ChatGPT said:

Your request for chat history is completely understandable, especially for those who want to revisit previous conversations, continue complex discussions, or reference past information. While I can’t implement that feature directly, your feedback is valuable.

I’d recommend sharing this feedback directly with OpenAI through their official channels or within the app’s feedback options. They often use user feedback to prioritize new features and improvements, so expressing your need for chat history could help influence future updates.

If there’s anything else on your mind or another way I can assist, feel free to let me know!

You said: Can I copy this whole dialog and save it as text? ChatGPT said:

Yes, you can easily copy the entire dialog and save it as a text file. Here’s how you can do it:

Select the Text: Click and drag your cursor to highlight the entire conversation. Copy the Text: Right-click on the highlighted text and select “Copy,” or press Ctrl + C (Windows) or Command + C (Mac). Paste the Text: Open a text editor like Notepad (Windows) or TextEdit (Mac), then paste the copied text by right-clicking and selecting “Paste,” or by pressing Ctrl + V (Windows) or Command + V (Mac). Save the File: Once the text is pasted, you can save the file by going to “File” > “Save As,” then choose your desired location and file name.

This will allow you to keep a record of the conversation for future reference.

You said: It only works for the latest answer. ChatGPT said:

If you’re experiencing issues with selecting and copying the entire conversation at once, here’s a workaround:

Use the Browser’s “Select All” Feature: On the web or in an app, try pressing Ctrl + A (Windows) or Command + A (Mac) to select all the text on the screen. This should highlight the entire conversation. Then, press Ctrl + C (Windows) or Command + C (Mac) to copy the highlighted text. Copy in Sections: If the “Select All” method doesn’t work, you might need to scroll up and copy the conversation in smaller sections. Highlight a section of the conversation, copy it, and paste it into your text editor. Then, repeat the process for the remaining sections. Browser Extension: If you’re using a browser, certain extensions or tools like “Evernote Web Clipper” or “Simplified Notepad” might allow you to clip the entire webpage or copy all text for easier saving.

Let me know if you encounter any issues, and I can try to guide you further!

You said: It still only works for the latest answer.

ChatGPT said:

If you’re only able to copy the latest answer, this might be due to the way the chat interface is designed, especially in some apps or browsers. Here’s an alternative approach you can try:

1. Screenshot the Conversation: If you can’t copy the text directly, you can take screenshots of the conversation. Most devices allow you to take a screenshot by pressing specific keys: Windows: Press Windows Key + Shift + S to select an area to screenshot. Mac: Press Command + Shift + 4 to select an area to screenshot. iOS/Android: Use the physical buttons on your device to take a screenshot. You can then save these screenshots as images for future reference. 2. Use the Browser’s “Print” Feature: If you’re on a browser, you can try printing the page to a PDF, which will capture the entire conversation. On your browser, press Ctrl + P (Windows) or Command + P (Mac) to open the print dialog. Choose “Save as PDF” instead of printing, and this will create a PDF file of the entire chat. 3. Use Developer Tools (Advanced Method): For advanced users: In a browser, you can open the developer tools (F12 or Ctrl + Shift + I on most browsers), inspect the elements of the conversation, and manually extract the text from there. However, this is a more technical solution.

If these methods don’t work or you have further issues, let me know, and we can explore other options!

The UI is a small fixed-size window (like those chat windows you see on service sites), so screenshooting made no sense. And the other two options weren’t worth trying.

But when I went to ChatGPT on the Web, there was my history outlined in the left panel:

The top one was the dialog I just had, displayed in the Web interface. The app could have @#$%^ told me that was a feature.

But it didn’t, because it’s lame. So far.

 

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. GitHub goes down, do Google’s Pixel team use Apple watches?, NVIDIA’s latest AI chip delayed, cybersecurity startup Lacework sells for a fraction of valuation, and more.

AWS su…

Read more

8月22日(木) 20時より、「そのQRコード、安全ですか？〜クロスデバイスの認証認可フローに潜む脅威と対策」と題して、秋田の猫こと @ritou さんをお迎えしてYouTube Live 配信を行います。

QRコードを使ってログインすることが日々増えてきています。でも、それって安全なんでしょうか？ 複数のデバイスをまたがった認証認可には落とし穴がたくさんあり、IETFでもベストプラクティスドキュメントの作成が遅々として1行われている状態です。

そこで、今回はこのあたりにお詳しい秋田の猫こと@ritou さんにお越しいただき、お話をしていただこういう企画です。[資料]

↓↓↓配信トラブルで新URLになっています↓↓↓

Genspark.ai 微妙に気になる点はあるものの、Perplexityに比べてかなり突っ込んだ回答をくれている。

以下、手を加えずにそのまま回答を転記します。どう思います？

なお、これを真に受ける人がいると良くないので、違うところは【←これは間違い】みたいに書いておきますね。

Gensparkの回答。

OpenID Connect (OIDC) についてまとめます。

OpenID Connectは、OAuth 2.0の認証・認可メカニズムを拡張したアイデンティティプロトコルです。2014年2月26日に最終仕様が発表され、現在ではGoogleやMicrosoftなど多くのIDプロバイダーによって広く採用されています4。

主な特徴 認証と認可の統合: OIDCはOAuth 2.0を基盤としており、認証と認可の両方をサポートします。特に、IDトークンを使用してユーザーの認証を行います124。【←正確にはユーザ認証の結果を連携します、だがまぁいいか】 IDトークン: IDトークンはJWT（JSON Web Token）形式で発行され、ユーザーの識別情報や認証結果を含みます。これにより、ユーザーのなりすましを防ぐことができます235。 シングルサインオン (SSO): OIDCはSSOをサポートしており、一度のログインで複数のアプリケーションにアクセスできるようにします14。【←SSOの定義にもよる】 多様なクライアントサポート: シングルページアプリケーション（SPA）やネイティブモバイルアプリなど、多様なクライアントで利用可能です4。 セキュリティ強化: OIDCは暗号化やデジタル署名を使用してトークンの改ざんを防ぎます。また、PKCE（Proof Key for Code Exchange）などのセキュリティ機能もサポートしています24。【←PKCEは別仕様。同一主著者によるものだが。】 フローの種類

OIDCには主に以下の3つのフローがあります24:

認可コードフロー: 最も安全性が高く、サーバーサイドアプリケーションに適しています。アクセストークンとIDトークンが直接返されないため、安全性が高いです。 インプリシットフロー: SPAなどで一般的に使用されるフローで、トークンがリダイレクトURIで直接返されます。 ハイブリッドフロー: 認可コードフローとインプリシットフローを組み合わせたもので、ネイティブアプリやバックエンドサーバーがある場合に適しています。 利点と用途 導入の容易さ: OIDCは簡単なコード変更で導入可能であり、GoogleやMicrosoftなど多くの企業で採用されています5。【←RP側は、ね。OP側は結構たいへんではある。複雑さをほとんどOP側に寄せているので。】 ユーザー体験向上: SSOによりユーザーは一度のログインで複数のサービスにアクセスできるため、利便性が向上します14。 セキュリティ強化: IDトークンやアクセストークンの使用により、不正アクセスや情報漏洩を防ぐことができます125。 柔軟な認証方法: ユーザー名/パスワード、SMSコード、生体認証など、多様な認証方法をサポートしています4。

OIDCは、OAuth 2.0の利点を活かしつつ、認証機能を強化したプロトコルとして広く利用されています。特にモバイルアプリやウェブアプリケーションにおいて、その利便性とセキュリティから重要な役割を果たしています。

ちなみに、Perplexityだとこんな感じ。 OpenID Connectの概要

OpenID Connectは、OAuth 2.0を基盤としたアイデンティティ認証プロトコルです。OAuth 2.0がリソースへのアクセス許可を管理するためのプロトコルであるのに対し、OpenID Connectはユーザーの認証を行うために設計されています。これにより、ユーザーが一度ログインするだけで複数のウェブサービスにアクセスできるシングルサインオン（SSO）を実現します[1][2][3]。

特徴とメリット シンプルな実装: OpenID ConnectはOAuth 2.0を拡張した形で設計されており、標準化された仕様に基づいているため、実装が容易です[2][3]。 セキュリティ: ユーザーの認証情報を一元管理し、複数のサービス間で安全に共有することができます。これにより、ユーザーの認証情報が漏洩するリスクを低減します[6]。 柔軟性: OpenID Connectは、ユーザーの同意のもとで、必要に応じて追加の属性情報（例：住所やメールアドレス）をサービス間で共有することが可能です[6]。 動作の仕組み

OpenID Connectは、ユーザーがあるサービス（例：WebサイトA）にログインする際に、別のサービス（例：WebサイトB）の認証情報を利用することを可能にします。以下はその一般的な流れです[8]:

ユーザーがWebサイトAにアクセスし、WebサイトBの認証情報でログインすることを選択。 WebサイトAはWebサイトBに対してIDトークンの発行を依頼。 WebサイトBがユーザーに認証を要求し、ユーザーが認証情報を入力。 WebサイトBがIDトークンを生成し、WebサイトAに提供。 WebサイトAがIDトークンを使用してユーザーを認証し、ログインを完了。 OpenID ConnectとOAuthの違い 目的の違い: OAuthは主に「認可」を目的としており、特定のリソースへのアクセス権を管理します。一方、OpenID Connectは「認証」を目的としており、ユーザーの身元を確認します[2][5][6]。 使用シナリオ: OAuthはAPIアクセスなどで利用されることが多いのに対し、OpenID ConnectはユーザーのログインやSSOの実現に用いられます[5][6].

OpenID Connectは、認証と認可を組み合わせたプロトコルであり、現代のウェブアプリケーションにおいて重要な役割を果たしています。特に、ユーザーエクスペリエンスの向上とセキュリティの強化に貢献しています。

Citations:
[1] https://solution.kamome-e.com/blog/archive/blog-auth-20221108/
[2] https://www.macnica.co.jp/business/security/manufacturers/okta/blog_20210901.html
[3] https://www.ibm.com/docs/ja/was/8.5.5?topic=au-openid-connect-overview-1
[4] https://www.openid.or.jp/document/
[5] https://www.okta.com/jp/identity-101/whats-the-difference-between-oauth-openid-connect-and-saml/
[6] https://www.financial.ctc-g.co.jp/itinfo/2402-oidc
[7] https://www.ibm.com/docs/ja/was-nd/9.0.5?topic=users-openid-connect-overview
[8] https://zerokara.dds.co.jp/oauth_oidc/oauth_oidc/

Classical pianist Jayson Gillham was pulled from an upcoming performance in Melbourne after dedicating a piece of music to the journalists of Gaza. The Melbourne Symphony Orchestra now says the cancellation was due to an error and they appear to be rescheduling the concert.

こんにちは、富士榮です。

毎回、IIW（Internet Identity Workshop）などのイベントと併せて開催されるOpenID Foundation Hybrid Workshopですが、次回のIIW（2024/10/29〜10/31）でも開催されます。

今回はCISCOのメンローパークのオフィスが会場です。

（前回はGoogleで開催でした。レビューはこちら）

ということでアナウンスです。

https://openid.net/registration-oidf-workshop-cisco-october-28-2024/

Thank you kindly to Cisco for hosting this hybrid workshop on Monday, October 28, 2024 12:30-3:45pm PT prior to IIW Fall 2024.

Cisco Campus

Bldg. 10 - Menlo Park

300 E Tasman Dr

San Jose, CA 95134

まだアジェンダなどは出てきていませんが、OpenID Foundationの各ワーキンググループの最新のアクティビティの確認ができるのでぜひ登録してみてください。

 

In mid 2024, GenAI tools powered by large language models (LLMs) are widely used by software engineering professionals, and there’s no shortage of hype about how capable these tools will eventually become. But what about today? We asked the software engineers who use them.

Our survey on this topic was filled out by 211 tech professionals; and we then analyzed this input to offer a balanced, pragmatic, and detailed view of where LLM-powered development tooling stands, right now. 

In Part 1 of this mini series we covered:

Survey overview

Popular software engineering AI tools

AI-assisted software engineering workflows

The good

The bad

What’s changed since last year?

In this article, we cover:

What are AI tools similar to? AI is akin to autocomplete, or pairing with a junior developer, or a tutor/coach, say survey responses.

State of AI tooling in 2024: opinions. There’s a roughly equal split between positive, neutral, and negative views on how useful AI tooling is for software development.

Critiques of AI tools. Many respondents are unimpressed by productivity gains, concerns exist about copied-pasted output, and other criticisms.

Changing views about AI tooling over time. After using the tools for 6 months or longer, most developers are either more positive, or slightly less enthusiastic than before. A minority feel more negatively.

Which tasks can AI already replace? Simple, constrained problems, and “junior” work and testing are tasks the tools can handle today. Plenty of engineers believe AI tools won’t fully replace any software engineering role or responsibility.

Time saved – and what it’s used for. Some respondents save almost no time, while others say AI tools create 20% more time for other things.

1. What are AI tools similar to?

In the survey, we asked professionals for their views on GenAI tooling, and several described it in metaphorical terms. Here’s a selection of answers to the question: “AI tooling is like __:”

… autocomplete. “As an addition to enhance developer productivity, it will become somewhat "invisible" and something we take for granted, like code auto-completion or grammar hints.” – CTO, 18 years of experience (YOE)

… pairing with a junior programmer. “Copilot is like pair programming with a junior programmer. I use it to do the rote work of filling in the details of my larger design.” – principal engineer, 45 YOE

… an eager intern. “It’s like having a really eager, fast intern as an assistant, who’s really good at looking stuff up, but has no wisdom or awareness. I used to think it would improve to quickly become a real assistant, but the usefulness of the tools haven’t meaningfully improved in the past year, so our current LLM approach feels like it might be petering out until we get a new generation.” – director, 15 YOE

… a tutor or coach. “It’s great as a tutor or coach, answering very specific questions quickly and painlessly. It gets you started or unblocked really well.” – distinguished engineer, 25 YOE

… an assistant. “For me it's more of an assistant which can help me in day-to-day tasks.” – architect, 9 YOE

… pre-Google search. “It is like when we didn't know which search engine was best and we had to multihome and try multiple different sites (Inktomi, Hotbot, Altavista, etc.) in the pre-Google days.” – data scientist, 20 YOE

2. State of AI tooling in 2024: opinions

We asked readers for your views on the current state of AI tooling, and the replies span the spectrum from positive to negative. Interestingly, the split is roughly equal between readers on this question:

Current state of AI tooling, based on 211 survey responses

Let’s dive into each group, starting with the upbeat.

Moderately positive views

The most commonly-mentioned benefits of AI tools:

Split within the ‘moderately positive’ feedback group

“Useful.” Lots of respondents say AI tooling is useful, but not particularly groundbreaking in its current form. A few quotes:

“Copilot is generally useful and can save a lot of time. Code completion systems have been around for years and this seems a natural evolution, if not a big step up. As an "old timer," I'm always a bit skeptical, but so far it's been a fairly good augmentation for my team.“ – Director of Engineering, 30 YOE

“It saves me half the time on a task, for about 10% of my tasks. I hope it improves significantly, and I am not worried about my job for the next 5 years." – software engineer, 17 YOE

“It works well, as the name implies, as a copilot. I feel it's sometimes quicker to ask a question than trying to hunt down an answer on Google, especially since copilot has some context from the workspace. It's also useful as an initial PR, even if I don't agree with or implement all its suggestions. For code generation, I found it more useful for writing unit tests than actual business logic.” – software engineer, 4 YOE

“It's a decent autocomplete. It’s good at answering simple questions when I'm working with a new language, and has basically replaced Stackoverflow for me. My expectations are that it will only get a little more powerful, and still only be useful for learning or simple completion, but the big value add will come from training on our specific codebase to enable things like generating the entire boilerplate for a test.” – software engineer, 13 YOE

“It’s super, super useful day to day, to speed up routine tasks and help with the empty whiteboard problem“ – principal engineer, 15 YOE

“It's pretty good. I give it 6 out of 10” – software engineer, 2 YOE

A software engineer with 7 years of experience gives their top uses for GitHub Copilot:

“Explain this to me” feature (most useful)

Chatbot integrating with company-internal documentation (useful)

Code generation (helpful if you know what to look for)

Autocomplete (on a par with older autosuggest tools, perhaps a bit more capable)

“A solid choice for prototyping and getting started.” Several respondents distrust the outputs of AI, but see its usefulness as a kickstarter:

Building a prototype for testing things

Putting together proof-of-concepts

Generating boilerplate code or scaffolding for a project

Getting started on a new project

Understanding a new codebase

A sounding board for questions

Several experienced engineers mention large time savings from using AI tools for prototyping, and building first proof of concepts which need not be production-ready, secure, or even correct. Quick and dirty, done fast; that’s the goal! AI tooling is helping greatly with this, says the survey.

“Good for learning and research.” Many people note these strengths:

Good at searching for relevant coding-related information

Helpful for explaining coding concepts, syntax, and use cases

Good for learning about technical and non-technical domains. A senior engineer uses it to learn about functional programming concepts and accounting, and says they learn faster than with textbooks

Helpful for looking up information on technologies and open source frameworks/libraries which the tool was trained on, or can access

“A great interactive rubber ducking / pair-programmer tool” Talking with the tool to get unstuck is mentioned by many devs. “Rubber ducking” is a problem-solving method of describing a problem to a real or metaphorical rubber duck; the idea is that vocalizing and explaining it out loud helps lead to a solution. 

Talking to an AI tool is different because it responds, unlike the duck. It can raise overlooked pointers and ideas. A staff engineer with 15 YOE calls it an “interactive pair programmer.” The same engineer also shares that it’s less helpful with anything more than simple pieces of code.

“I can see it evolving further.” Several respondents say there’s plenty of room for improvement. More integration touchpoints is an area which a CTO with 20 YOE hopes to see more progress in:

“Currently it can be a good pairing buddy for common languages/frameworks that don't change too often. In future, would be great to see some more integrations - i.e. suggestions for code improvements from IDE, ability to dive into projects and provide feedback on alternative approach from design perspective, tests auto-generation based on specs and existing codebase"

“Still learning how to use it.” It takes time and experimentation to figure things out. Software engineer Simon Willison (25+ YOE, the creator of the Django framework) has publicly said it took him quite a while to learn how to use GitHub Copilot productively. Meanwhile, a backend developer with 6 YOE shares that they are doing the same:

“I am not intimidated by AI technologies. On the contrary, I am trying to incorporate such tools into my work as much as possible. I feel I should invest more time in learning how to use AI tools. The generation of tests could potentially be improved. Current tools such as Copilot and CodiumAI have limitations for load context and do not often utilize existing fixtures by Pytest. I still write complex business logic manually, and often make corrections by hand because the AI cannot do it, or I’m too lazy to debug the prompt." 

Very positive views Split within the ‘very positive’ feedback group

“Translating” between programming languages works great. Several developers share that a useful use case is giving the AI tool code in one language to be translated into another. Thanks to the architecture of LLMs, they translate well between programming languages, and also human languages. An engineering manager (25 YOE) says:

“It’s great for taking an idea prototyped in one language (bash), and translating it into another language (python). I can run both programs and assume that they return the same output.”

“It’s great!” Plenty of responses are strongly positive. Here are some from the survey about Copilot and ChatGPT:

Works very well on languages less used by developers

An “invaluable autocomplete+++,” as shared by a senior software engineer, 10 YOE

“Understands the codebase and quickly catches up to its style” - software architect, 25 YOE

Helps figure out issues with toolchain and development environments - senior engineer, 20 YOE

Wonderful for generating unit tests for classes - staff engineer, 20 YOE

A chat AI assistant is the first recourse when stuck on a problem - senior software engineer, 7 YOE

“Game changer.” A sizable number of responses are bullish about the productivity gains of the tech today. The quotes below come from professionals with at least 10 YOE, and plenty of hands-on experience:

“GitHub Copilot has been a game changer for day-to-day productivity in our company. GPT-4 is also widely used for documentation, meeting notes, and requirements gathering. I assume it will only get better, and become more effective at reducing toil and helping teams produce better software, particularly with the non-core-coding tasks (documentation, unit tests, sample code and starter kits, templates, etc).” – staff engineer, 30 YOE

“It’s an enormous productivity boost, even with existing tools. I expect it to improve over time, though not necessarily by orders of magnitude.” – principal engineer, 25 YOE

“It’s essential for any modern engineer to have AI tooling in their stack. Otherwise, it’s a massive productivity hit.” – engineering team lead, 12 YOE

“Game changer. Makes writing test cases really easy. Shifts most of the coding I do to thinking/planning from writing. I love using it to write security policies, too!” — Director of security engineering, 12 YOE

3. Critiques of AI tools Split among ‘moderately critical’ respondents “Unimpressed”

The single most-cited criticism is that AI tools are not living up to their promise. Complaints from active users included:

Stagnation. The top gripe. LLM technology hasn’t changed in more than a year, since the release of ChatGPT 4.0, and it’s not getting better. Approaches like RAG can only bring so much improvement, according to a software architect of 15 years, and one with 20 YOE.

GitHub Copilot is the “only” tool that works. Several people say Copilot is the only tool they’re not entirely dissatisfied with, after testing several. However, these developers add that Copilot gets suggestions wrong as often as it’s right, and that unless you’re in a rush, it’s better to write code without automated suggestions.

Useful for simple stuff, poor at more complex tasks. It’s good at speeding up simple-enough refactoring, tedious use cases and “routine” tasks like config file changes, and working with regular expressions. But it doesn’t help much when working across a complex codebase, especially one that follows less common coding or architecture patterns.

Limited use cases. Outside of repetitive changes and boilerplate generation it’s not useful because it introduces hard-to-spot bugs.

Too much effort. Outside of code suggestions, it’s a lot of effort to get something complex that works as expected.

Unreliable. You need to stay on your toes because just when you start to trust it, it generates non-working, buggy code.

A “fancy autocomplete.” This is the view of an engineering team whose members have used the tool for around 6 months. Hallucinations are why it’s not trusted more.

More hindrance than help. A staff engineer with 10 YOE is correcting AI output more than benefiting from it. They expect things to improve, but see little value in the tooling as it stands.

Tedious. It’s more work to bulk-generate tests, and tools can stumble when faced with untypical software architecture. A software developer (9 YOE) expects things to improve when the tool can “digest” complete projects.

“Even the boilerplate is not what I need.” A senior software engineer with 5 YOE voices disappointment that instructed boilerplate code ended up doing the opposite of what was intended. For context, this engineer is just getting started on AI tools; perhaps working on prompts will improve outputs.

Concerns about engineers copy-pasting AI output 

Half a dozen engineers with more than 10 years of experience voiced the concern that less experienced colleagues copy-paste code output from AI tools, without knowing what it will actually do. Seasoned developers list these gripes about:

Forgetting that AI-generated code is unverified. “Engineers copy-pasting code is nothing new, it’s existed since the StackOverflow days. However, StackOverflow has a human touch and verification; AI has neither.” (principal engineer, 11 YOE).

Trusted as a source of truth, but isn’t one. “It works well if you have enough knowledge to understand the output and to challenge it. It’s very dangerous for junior devs as they tend to use it as a source of truth, which it is not, at least not today.” (software engineer, 6 YOE)

Doesn’t truly help newer devs: “These tools are directionally helpful for seniors who can identify errors, but detrimental to juniors who won't know the difference.” (AI practice lead, 17 YOE)

Degrades people’s problem-solving skills. “I worry the younger generation will become dependent on LLMs, and that their own critical thinking and problem-solving skills will diminish.” (DevOps engineer, 8 YOE) I’d add that older generations tend to underestimate how the next generation adapts. Our recent GenZ survey in this publication showed the new generation of talent is very capable; so we might need to worry less!

Fair Criticism related to the architecture of LLMs

The below criticism is about the inevitable nature of LLMs, thanks to their architecture. LLMs are trained on a large corpus of textual training data, and then generate the next most likely token based on an output. Thanks to this architecture, they will perform worse when encountering scenarios that were not part of their training data, and are prone to hallucination. We previously covered How LLMs like ChatGPT works – as shared by the ChatGPT team.

“Doesn’t work well on custom tech stacks and novel architectures.” Experienced engineers say that current tools can’t get context on an entire code repository, or on all projects within a company. This lack of context gets in the way of helpful output for projects with custom code structures and coding approaches.

Here’s the problem according to a staff engineer (30 YOE):

“The killer features for us would be a broader understanding of our ecosystem of APIs and libraries, to recommend common standard approaches within our company when writing new code or building new products.”

This criticism makes sense given how LLMs are trained; most training data is open-sourced code. At the same time, technology used at larger companies is often home-grown, and so these tools need additional context to work optimally. This context could be provided via RAG (we covered more on RAG.) We also previously covered how LLMs work and are trained.

“Doesn’t use context beyond file-level.” A valid criticism is that current LLM tools focus on the file level, not the project level. Several developers voice hope that AI tooling shall be reimagined to work at project-level, and that AI assistants will get additional context to the single open file.

A software developer using these AI tools for more than two years says:

“There is not yet a good tool with the speed, quality and developer experience of GH copilot, which is also aware of the full project context (types, interfaces, overall patterns, file structure,) and that we can ask questions to without leaving the IDE. I’ve tried GH Copilot X, which is an attempt at this, but it's definitely not there yet and has been a net negative.

I would pay for a tool that solves this problem smoothly. At the same time, I wouldn't conform with anything less than GH Copilot in terms of speed and quality of its autocomplete.”

And a principal engineer with 30 years of experience puts it like this:

“Copilot is good at very small, self contained tasks, but those are relatively rare. It’s bad at anything that needs broad context across the codebase.”

“Need to learn how to use it.” Several developers share that they had to invest a lot of time and effort in getting the tool to be useful. Some observations:

You need to guide it, akin to helping an inexperienced engineer (staff engineer, 8 YOE)

It needs a lot of handholding and domain expertise to make it useful (software engineer, 9 YOE)

It can be hard to ask the “right” question, and sometimes it’s easier to just skip it (software engineer, 13 YOE)

You need to have specific expectations, e.g. when generating tests, or how to have it work with data (senior software engineer, 25 YOE)

Hit and miss. “Sometimes it's like it's reading my mind, sometimes it's just so totally wrong it outweighs any benefits.” (SRE, 15 YOE)

A staff engineer with 25 years YOE gives an excellent summary:

“It's extremely useful if you understand its limitations. I get the best results when I use it to fill in the blanks. 

Asking it to write a whole class, or an entire test file is dangerous because it'll give you something that looks right. However, when you dig in, there’s all kinds of things that just won't work. [But] it's gotten noticeably better since the early days. 

I trust it enough now that if I lay out the code – set up the class name, important method name, and parameters – then I can just kinda tab through the details while double checking each chunk it gives me. Having good accompanying tests is also important.”

“Hallucination still a problem.” Several engineers raise this ongoing issue. Here’s a solutions architect with 15 YOE’s take:

“Sometimes it hallucinates methods that don’t exist. I’ve used trials of several services, and haven’t seen long-term improvement in any one."

“Doesn’t work well for brainstorming.” A systems engineer finds AI tools are a poor choice for brainstorming because they can generate lots of ideas, but its detailed descriptions legitimize bad ideas! Also, the tool cannot tell what’s useful for a business.

“Lowered my expectations as a less experienced developer.” A few devs share that over time AI tools seem to have gotten less reliable, or to not work as hoped. Interestingly, this observation comes from developers with less than 3 years of experience. 

Perhaps this could be a case of the tech being at a point where it’s hard to judge when the output is not what’s needed. Or is it that more experienced developers come to every new tool with low expectations because they know that reality hardly ever lives up to hype?

Very critical

Read more

I decided to do a bit of practice in prompt construction and with a spreadsheet of example software release data. I was curious how well ChatGPT would perform using an Excel spreadsheet of data points. I had seen examples of folks querying across a spreadsheet but you never knew how contrived the experience might be. I decided to see for myself with a limited dataset how it performed. From this simple dataset it exceeded my expectations, which were low, but it would have exceeded them had they been much higher.

The Excel spreadsheet was just a very small set of example data I had set up to create pivot charts and reporting for software release metrics. It contained only 9 rows of about 30 data points each. Four of the columns titled “is release delivered”, “is hotfix”, “is planned release”, “was rolled back” represented with values of 1 or 0 existed with other data points such as deployment duration, hotfix lead time and, as mentioned, a few dozen more fields.

I started with the prompt: “The attached spreadsheet contains release management and deployment data points. It contains metrics such as if a release was successful, if there were rollbacks, the number of hotfixes etc. Using the data how many rollbacks were there? If you don’t know just say you don’t know.” It came back with 1 roll back, which was correct. Next up was to ask it for percentage of planned releases as opposed to hotfixes and rollbacks. Again correct. I asked it for something I can’t get out of a pivot chart but which is easily calculated given a rang of data. “what is the median deployment duration?” Correct, mostly. It said the duration was in days. There are no unit of measure on the data itself but one tab of the spreadsheet contained a data dictionary that said the deployment duration was in hours. It didn’t pick up on that. I asked for a chart of min., max, avg., median values for hotfix lead times. Nailed it. I asked for a modification of the chart to show the values above the bar of each data element. All good.

Time to give it something a bit more difficult — “compared to any industry averages are the hotfix to planned release ratio good?” I thought a bit and came back with the below and more, but for the core of the response it was:

Industry Standards: The acceptable ratio of hotfixes to planned releases can vary significantly depending on the industry, the complexity of the software, and the release cadence. In some industries, a lower ratio is ideal as it indicates better initial quality and stability. In fast-paced industries with frequent releases (e.g., agile environments), a higher ratio might be more common.
Benchmarking Data: Industry benchmarks often suggest that a well-functioning release process should have more planned releases than hotfixes. For instance, in mature organizations, the ratio of hotfixes to planned releases might be around 1:5 to 1:10, depending on the complexity and maturity of the development and testing processes.
Your Ratio: A ratio of 0.6 indicates that there are fewer planned releases compared to the number of hotfixes, which may be higher than industry averages, particularly in environments where stability and quality are prioritized.

Where it came up with the benchmarking data ratios I have no idea. I suspect it is just as likely a hallucination as based on something. I’ll likely never know. I tried to do some searches for such data and came up empty but it doesn’t mean it’s not out there somewhere implied, derivable, or otherwise. Regardless, it came up with the right idea of how the relationships should be.

Simple data, simple questions but good results. All in all I was pleased.

This post shows how to implement a Swagger UI using a .NET 9 produced OpenAPI file. The Swagger UI is deployed to a secure or development environment and is not deployed to a public production target. Sometimes, it is required to deploy the Swagger UI to a development deployment target and not the test or the production deployments. The security headers need to be weakened to allow the Swagger UI to work.

Code: https://github.com/damienbod/WebApiOpenApi

Setup

The post uses the OpenAPI Json created by a .NET 9 ASP.NET Core application. See this blog:

Implementing an ASP.NET Core API with .NET 9 and OpenAPI

Two further packages were added to this project, one for the generation of the Swagger UI and the second package to generate the required security headers for an API using JWT Bearer tokens.

Swashbuckle.AspNetCore.SwaggerUI NetEscapades.AspNetCore.SecurityHeaders

The ASP.NET Core API application has already generated the OpenAPI definitions in a Json file. The Json can be used to create the UI. If the application is deployed to production, the Json file is not created and the security headers are deployed with the most restrictions. If the application is deployed for development, the Json is deployed and the security headers are weakened to allow this to work.

// Open up security restrictions to allow this to work // Not recommended in production //var deploySwaggerUI = app.Environment.IsDevelopment(); var deploySwaggerUI = app.Configuration.GetValue<bool>("DeploySwaggerUI"); app.UseSecurityHeaders( SecurityHeadersDefinitions.GetHeaderPolicyCollection(deploySwaggerUI)); // ... other middleware app.MapOpenApi("/openapi/v1/openapi.json"); if (deploySwaggerUI) { app.UseSwaggerUI(options => { options.SwaggerEndpoint("/openapi/v1/openapi.json", "v1"); }); }

The DeploySwaggerUI configuration is used to specify if the deployed version supports both a UI and an API or just an API with the most restrictive security settings.

{ // Open up security restrictions to allow this to work // Not recommended in production "DeploySwaggerUI": true,

Setup security headers

The security headers are setup so that if the deployment is for development, scripts and styles are allowed. The configuration allowing scripts is weak and not recommended for production.

namespace WebApiOpenApi; public static class SecurityHeadersDefinitions { public static HeaderPolicyCollection GetHeaderPolicyCollection(bool isDev) { var policy = new HeaderPolicyCollection() .AddFrameOptionsDeny() .AddContentTypeOptionsNoSniff() .AddReferrerPolicyStrictOriginWhenCrossOrigin() .AddCrossOriginOpenerPolicy(builder => builder.SameOrigin()) .AddCrossOriginEmbedderPolicy(builder => builder.RequireCorp()) .AddCrossOriginResourcePolicy(builder => builder.SameOrigin()) .RemoveServerHeader() .AddPermissionsPolicy(builder => { builder.AddAccelerometer().None(); builder.AddAutoplay().None(); builder.AddCamera().None(); builder.AddEncryptedMedia().None(); builder.AddFullscreen().All(); builder.AddGeolocation().None(); builder.AddGyroscope().None(); builder.AddMagnetometer().None(); builder.AddMicrophone().None(); builder.AddMidi().None(); builder.AddPayment().None(); builder.AddPictureInPicture().None(); builder.AddSyncXHR().None(); builder.AddUsb().None(); }); AddCspHstsDefinitions(isDev, policy); policy.ApplyDocumentHeadersToAllResponses(); return policy; } private static void AddCspHstsDefinitions(bool isDev, HeaderPolicyCollection policy) { if (!isDev) { policy.AddContentSecurityPolicy(builder => { builder.AddObjectSrc().None(); builder.AddBlockAllMixedContent(); builder.AddImgSrc().None(); builder.AddFormAction().None(); builder.AddFontSrc().None(); builder.AddStyleSrc().None(); builder.AddScriptSrc().None(); builder.AddBaseUri().Self(); builder.AddFrameAncestors().None(); builder.AddCustomDirective("require-trusted-types-for", "'script'"); }); // maxage = one year in seconds policy.AddStrictTransportSecurityMaxAgeIncludeSubDomains(maxAgeInSeconds: 60 * 60 * 24 * 365); } else { // allow swagger UI for dev policy.AddContentSecurityPolicy(builder => { builder.AddObjectSrc().None(); builder.AddBlockAllMixedContent(); builder.AddImgSrc().Self().From("data:"); builder.AddFormAction().Self(); builder.AddFontSrc().Self(); builder.AddStyleSrc().Self().UnsafeInline(); builder.AddScriptSrc().Self().UnsafeInline(); //.WithNonce(); builder.AddBaseUri().Self(); builder.AddFrameAncestors().None(); }); } } }

When the UI is deployed to in development mode, the Swagger UI is displayed and the user can enter a valid access token and use the APIs.

Notes

At present, the Swagger UI does not support script best practices and allowing this to work means deploying an unsecure web application with weak security. Any UI web application should use a strong CSP definition, for example like CSP nonces. An API has no UI and so should be locked down.

Links

https://github.com/martincostello/dotnet-minimal-api-integration-testing

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/minimal-apis/aspnetcore-openapi

https://learn.microsoft.com/en-us/aspnet/core/web-api/action-return-types

https://github.com/RicoSuter/NSwag

https://swagger.io/

https://swagger.io/specification/

“Some argue that by aggregating knowledge drawn from human experience, LLMs aren’t sources of creativity, as the moniker “generative” implies, but rather purveyors of mediocrity. Yes and no. There really are very few genuinely novel ideas and methods, and I don’t expect LLMs to produce them. Most creative acts, though, entail novel recombinations of known ideas and methods. Because LLMs radically boost our ability to do that, they are amplifiers of — not threats to — human creativity.” – How LLMs Guide Us to a Happy Path for Configuration and Coding

Part of the LLM series at The New Stack.

At this year’s, DWebCamp, my friend and collaborator Kaliya aka IdentityWoman and I ran a session titled:

“Envisioning the future: what if a single, decentralized, interoperable, open social network indeed emerges and wins over the proprietary platforms?”

We did something that, at least for me, was new. We started with:

Assume it has happened. We have won. Like on page 1 of a sci-fi story, you wake up in the morning one day and the world you want is has suddenly arrived. At least 4 billion people interact with each other socially on an open, decentralized social network. All of the proprietary platforms have much smaller user numbers or have disappeared entirely; they have stopped mattering. Out of scope: how we get there, or which specific products/standards/technologies are being used.

We asked the question: “Describe what you see”. People who came to our session had a few minutes to meditate on that question, and then wrote down their thoughts, one at a time, on some stickies. They then announced what they wrote and put it on a big sheet of paper. Below is the electronic version of this paper (minus some items which I could not manage to decipher when transcribing):

Community focused and developed by the community Hate free zone Accessible resources (open source) for development by anyone Message flows automatically (I don’t have to decide who are the recipients) biofi.earth for decentralized technology Not harmfully powered Access to all medical data. Do I want all measurements? Which doctor may see what? How do I manage that? Serependipity / surprise me Anti-filter bubble Multi-modal “of one universe” Everything is forkable (in the open-source sense) Everything is fully transparent Attention management / needs to manage Personal and collective sense-making bubble Stigmergy / people marking How to make the network safe? Non-screen-centric “Netflix”-like subscription for all: distributed to builders / operators New funding models for open source Cooperative Attention Economy Fed by opt-in-able data sets Ability to opt-in/opt-out Personal digital exoskeleton / voice-centric Not one systems. Highly fragmented (needs are specific) A shared collaborative universe, not just “apps” Social “digital twin” objects Who you are is what you consume: food and information Photo gallery If it isn’t worth looking at for at least 10 seconds, it isn’t worth any time at all Tap to dive in, call, play, study / zoom out to explore and connect Non-repetitive Profile: Skills Expanse Reviews Map Faves Music Corporate news is dead, replaced by emergent editorial boards Threats community mapping / fear

What surprised me most is that there were so many comments that wanted to have a single “social universe” into which various apps and screens and voice are “windows”", rather than a single app. That’s very close to the original vision for the world-wide-web, and one that we’ve largely lost with all the apps we are using every day. But perhaps we can get it back?

Addition 2024-08-17: two more, which I don’t know how to transcribe.

Following the recent spate of work drafting RFCs for a binary distribution format and Meta Spec v2, and bearing in mind the vote to implementing PGXN v2 apps in Rust, over the last few weeks I’ve switched gears to write some code.

First I wrote JSON Schemas for the v1 spec and then wrote a full test suite using the boon crate. Next I wrote and tested JSON Schemas for the v2 spec. This process informed revisions to the RFC, uncovering inconsistencies and unnecessary complexities.

With META.json file JSON Schema validation fully worked up, I decided to work the functionality into a proper Rust crate and CLI to eventually replace the old PGXN::Meta::Validator Perl module and its validate_pgxn_meta CLI. This turned out to be a decent Rust starter project, requiring a fairly simple crate and CLI, but also allowed me to develop patterns to build and release binaries for a variety of platforms and architecture.

As a result, I’m happy to announce the release today of the pgxn_meta crate and pgxn_meta CLI v0.1.0, available for download on these platforms, thanks to cross and houseabsolute/actions-rust-cross:

darwin-amd64 darwin-arm64 freebsd-amd64 freebsd-i686 illumos-amd64 linux-amd64 linux-arm linux-arm64 linux-i686 linux-powerpc linux-powerpc64 linux-powerpc64le linux-riscv64 linux-s390x linux-sparc64 netbsd-amd64 solaris-amd64 solaris-sparcv9 windows-amd64 windows-arm64 windows-i686

Download the archive file appropriate to your platform, decompress it, and put the pgxn_meta (or pgxn_meta.exe) binary in your path. Or use the universal binary installer (ubi) to install it:

ubi --project pgxn/meta --exe pgxn_meta --in ~/bin

And of course you can use cargo to compile it from source:

cargo install pgxn_meta

Usage is simple: just run pgxn_meta in a directory containing the META.json file to validate:

❯ pgxn_meta META.json is OK

And optionally pass it the name of the file, as in this example parsing a test file with no version property:

❯ pgxn_meta corpus/invalid.json Error: "corpus/invalid.json jsonschema validation failed with https://pgxn.org/meta/v2/distribution.schema.json#\n- at '': missing properties 'version'"```

That’s it!

What’s Next?

Now that I’ve implemented validation and figured out multi-platform binary support for Rust apps, my next tasks are to:

Implement a pattern to convert a v1 META.json to the v2 format Create a pattern to merge multiple META.json files into one Write code to build PGXS extension into trunk packages Develop patterns to satisfy third-party dependencies for multiple platforms

Should keep me busy for a few weeks. Updates as I have them.

More about… Postgres PGXN JSON Schema Rust Metadata

The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.

Today, we cover:

Industry pulse. Court rules Google broke antitrust law, Twitter/X quits San Francisco, Google kills Chromecast, does NVIDIA trains model on Netflix and YouTube videos?

Intel’s rough business outlook and major reorg. It’s a third consecutive year of falling revenue at Intel. The company is reorganizing itself to align with the “foundry model” of the chip-making industry.

Founders of well-funded AI startups keep leaving for Big Tech. Character.ai is the latest AI company founded just 2-3 years ago, to be hollowed out by a brain drain to Big Tech. Is it a sign that AI startups will struggle to monetize, grow, or raise funding?

Buggy app to take Sonos 6 months to fix. The new Sonos app is slow, full of bugs, and unreliable. It took the company 3 months to acknowledge the problem, and now all hands are on deck to fix it. It’s valid to ask how this app ever reached customers.

Fallout of CrowdStrike’s global outage. The security provider is blamed for the mother of all software crashes, last month. But there were several prior cases of operating systems being broken at the kernel level. Now Delta Air Lines is suing for $500M, the reputational damage is vast, and compensation payouts could be devastating.

1. Industry pulse Google broke law by paying to be default search engine

The US Department of Justice (DOJ) sued Google, alleging the search giant broke antitrust laws by paying smartphone and browser vendors to be the default search engine on their products and devices. In 2021 alone, Google paid a combined $26B to Apple, Firefox, and other vendors for this, most of which ($18B) went to Apple.

A judge has now ruled the agreements were unlawful, stating:

“Google’s monopoly power, maintained by the exclusive distribution agreements, has enabled Google to increase text ads prices without any meaningful competitive constraint.”

There will be a hearing in September to decide what changes to its practices will be mandated for Google. An appeal by the tech giant looks inevitable; but as things stand, it will no longer be able to pay to be the default search engine on other platforms.

It’s a notable victory for the US regulator, proving that Google has used illegal means to maintain a search monopoly, with the consequence of stifling competition, while increasing its own ad pricing and revenue. It’s currently an open question as to what Google’s argument will be against this judgment – because the ruling appears reasonable and balanced.

Twitter/X quits San Francisco

Read more

The Tembo Blog published a post by yours truly last week about when to preload shared libraries and when not to:

Recently I’ve been trying to figure out when a Postgres extension shared libraries should be preloaded. By “shared libraries” I mean libraries provided or used by Postgres extensions, whether LOADable libraries or CREATE EXTENSION libraries written in C or pgrx. By “preloaded” I mean under what conditions should they be added to one of the Shared Library Preloading variables, especially shared_preload_libraries.

The answer, it turns out, comes very much down to the extension type.

I view this post as a kind of proto-chapter for an imagined book about developing extensions that I’d like to work on someday. I learned quite a lot researching it and responding to extensive feedback from more knowledgeable community members. It resulted in updates to the PGXN Meta preload property that I hope will inform binary distribution in the future. More on that soon.

More about… PGXN Postgres Extensions Preload Extensions Book

👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get articles like this in your inbox, every week, subscribe:

Subscribe now

The Pragmatic Engineer is back after a week-and-a-half’s summer vacation. Things are usually slower during this summer break period – recruitment included – since recruiters and candidates take holidays, too! Also, headcount for the first half of the year (H1) has usually been filled by now, while H2 headcount is often finalized during August, with hiring kicking off in earnest in September.

But this summer has been different. A subscriber based in New York pinged me, sharing what they’ve seen:

“As a software developer, I've noticed a slight uptick in recruiter activity in the last month or two. I initially dismissed it, but another developer mentioned noticing an increase in LinkedIn recruiter mail, primarily from smaller startups. I was wondering if you've heard of this, [and] if you'd do a piece shedding light on this phenomenon and its causes.”

This piqued my interest and I asked around, messaging more than 40 tech professionals. This revealed an interesting phenomenon: recruitment for software engineers and related disciplines did spike in June and July, defying the conventional pattern.

We cover this topic in this issue, addressing:

US uptick. Interest from recruiters up on the East and West Coasts.

More UK and EU inbounds. The uptick in Europe is especially interesting, given the summer holidays are longer there, and recruitment is typically lower at this time.

Why an uptick now? Hiring managers and executives share why their companies are hiring more. There are several reasons.

Product and design. Some uptick in recruiter interest for these areas, too.

Reality check. Some companies are not hiring more, and plenty of engineers aren’t seeing more responses.

Appendix: other regions. Data points from the Nordics, Central Europe, India, Latin America, and Israel.

1. US uptick

Below are some personal reports from engineering managers, software engineers, and data/ML engineers, who have observed the recruitment uptick. All shared their observations directly with me, and the responses are anonymized.

US East Coast

Early-stage startups, AI startups, and hedge funds seem to have accelerated hiring in the past few months, based on readers’ first-hand accounts:

An engineering manager (EM) at Big Tech, 10 years of experience (YOE), based in New York City (NYC):

“I have noticed an uptick in recruiting messages over the past month. These are mostly from early startups looking to hire for Head of Engineering positions, or AI startups looking for an Engineering Manager or Tech Lead. Also, it seems like finance headhunters are unusually active at the moment.”

Tech Lead, 8 YOE, NYC:

“I've been receiving an uptick in recruiter messages related to AI and scaleups. Even more of the ‘usual’ hedge fund recruiters have been active in the past few months.”

Senior ML Engineer, startup, 7 YOE, Boston:

“I've been noticing inbounds ratcheting up since April to the point that I had to turn off my ‘open to work’ settings on LinkedIn. I am still getting a decent amount of inbound, even with the setting turned off. It's definitely not like the peak late-2021-2022 frenzy (something I'd be surprised to see again) but it’s similar to – maybe higher than – the pre-Covid status quo.

My inbounds are largely from startups, but most of my experience is working in early stage startups, so there is a potential confounder in me!”

Senior Engineer, startup, 5 YOE, NYC:

“I’ve noticed that in the past month, there have been relatively more recruiters reaching out. As context, I've been looking for jobs since May. At the same time, out of all of the applications I've sent, only one company started the interview process with me.”

Software engineer, ex-Big Tech, 10 YOE Seattle: 

“I got laid off in May. Since then, I have been reached out to by recruiters from Amazon, Bytedance, and various consultancies. After I upgraded to Linkedin Premium, there has been an uptick in reach-outs from recruiters on Linkedin. I’m still unsure if the uptick could be attributed to this change, or if the market is getting better.”

Companies are seeing offer acceptance rates dropping, which is one indicator that it’s a bit more of an “employee’s market” than we’ve been used to in the recent past. From a director of engineering at a publicly traded tech company in Boston:

“While our hiring slowed slightly, our rate of hiring has been relatively steady over the past 2 years. For experienced roles, 6 months ago our offer acceptance rates were close to 100%, now they're at 75% or lower, for roughly the same offers, compensation-wise. Definitely feels like a thaw in the job market to me.”

US West Coast

AI startups, startups, and mid-sized tech companies, seem to be increasing their recruitment pace. Readers’ observations:

Senior software engineer (ex-Big Tech), at a scaleup, 10 YOE, San Francisco (SF):

“There has been an influx from AI startups for sure, and some regular startups, but I am now seeing mid-size and large companies reaching out too, with very few public companies reaching out.”

Senior engineer, publicly traded tech company, SF, 8 YOE:

“I've left my profile as "Open to work" for the last 4-5 months. This month (July), I've gotten a significant amount of inbounds from internal and external recruiters:

Inbound recruiter messages to a senior engineer at a publicly traded tech company in SF

Tech lead, Big Tech, 10 YOE, SF:

“My observation on the hiring uptick is that it isn’t just startups / scaleups. I'm starting to get more messages from the likes of Airbnb and Snowflake.”

Software engineer, 2 YOE, Los Angeles (LA):

“I had a recruiter from a startup reach out right after I announced my promotion, and another one via direct email. As someone with less than 2 years experience, that was very encouraging.”

Rest of the US

Startups and some mid-size and larger tech companies have been reaching out more, nationwide.

Senior frontend engineer, Big Tech, 15 YOE, western US:

“I have seen an uptick — and not just from startups! I've had reachouts from Datadog and TikTok in the past month. A few more than usual from my local market as well (Utah, which has a budding tech economy) that are higher quality than I normally see. Hopefully, this is evidence of a normalizing tech job market! “

Senior software engineer, startup, 8 YOE, midwest US:

“Recruiter messages to me on LinkedIn have absolutely picked up. I noticed because my inbox was pretty dead for the last year. Mostly startups, but 1 friend also sent me a job description for a position at GitHub as well.”

Director of data science, consultancy, 7 YOE northwest US:

“I have had an uptick in legitimate outreach. A few months ago it was dry, but it has suddenly gotten better.”

Lead engineer, startup, 6 YOE, western US:

“I categorized reach outs into three categories:

“Relevant” jobs are exactly my niche or current stack. 

“Almost relevant” jobs are outside my niche, or related to stacks I've used in the past. 

“Irrelevant” are totally off-base stacks, or things that look like spam/mass messages.

Here’s the “relevant” reach outs this person has received in 2024:

Relevant recruiter messages to a full remote, lead engineer at a startup, western US

Software architect, publicly traded company, 15 YOE, southern central US:

“I’m getting a lot of requests for interviews on LinkedIn lately. My inbox had been quiet for a long time, I probably received more recruiters messages in the last couple of weeks than the last 4 months.”

Software engineer, 2 YOE, southwestern US

“It's been pretty much a drought for the last two months; about one message a month. I did get a message from a new founder last week which led to a good conversation, though he won't be making any hiring decisions for at least another month. It sounded like he got a decently large block of funding, but is being extra cautious about where and how he deploys it.

In the past, namely 2021-2023, I was getting multiple messages per week, sometimes per day.”

2. More UK and EU inbounds

Recruiters in Europe take longer summer vacations than in the US, which makes the increase in recruitment activity across Europe eye-catching. Here’s what I’ve heard from tech professionals in these markets.

UK

Startups, finance companies, scaleups, and AI companies, seem to have increased recruitment reach outs in the last few months, according to some first-hand accounts:

Engineering manager, Big Tech, 15 YOE:

“I have observed an increase in reach outs from recruiters.

At work, I now also get a higher number of interview requests popping in my work calendar: 2-3 a week, vs 2-3 a month, 6-12 months ago. These interviews are all for L5+ (senior-and-above levels.)

I suppose we’re perhaps seeing the green shoots after a bleak 18-24 months. Or it might be temporary due to ‘AI hype’ (my company is investing in AI, big time!) Time will tell!”

Software engineer, publicly traded tech company, 10 YOE:

“The recruiter messages never dried up – they’ve been coming fairly regularly and (at least here in London) they’ve been a mix of startups, finance companies, and the occasional pre-IPO scale up. The only thing missing is Big Tech – save for Meta and Amazon, both of which have been fairly active.”

Staff software engineer, scaleup 15 YOE:

“I can confirm that I’ve seen an uptick in recruiter messages and with lower importance connection requests. I generally haven’t put effort into my LinkedIn profile, so for the past year it’s been rather quiet. Recently, I’ve gotten noticeably more ‘template’ recruitment messages than before.”

Senior backend engineer, 10 YOE:

“I’m seeing way more recruiter inbound activity over the past month or two. Mostly in fintech and climate tech in my case. Inbound is still limited to hybrid, or fully in person roles, which is acceptable to me. Willingness to sponsor visas still seems to be lacking in the inbound roles, which indicates to me there’s still some softness in the UK market.”

Senior ML Engineer at a Big Tech, 8 YOE:

“I have seen a huge uptick in reach outs from recruiters. This is largely smaller companies, but also includes top tier companies recently including OpenAI, Anthropic and Jane Street.”

ML engineer, publicly traded tech company, 5 YOE:

“There is an absolute uptick of recruiter inbound messages. In the last two months, the job specs have suddenly started flowing again. This is particularly true in my area of work in the MLOps / ML Platform space. The market for that is better than it was before the end of zero percent interest rates, mainly because it's a new skill market.”

Senior data scientist, 5 YOE:

“I have my profile as ‘not looking’ and had several approaches in the last few weeks (above normal.)”

Western Europe

Driving the hiring uptick seems to be startups and scaleups raising funding, mid-size companies, and some larger companies. Observations from a senior engineer, 8 YOE, Switzerland:

“I got around 4-5 messages from various recruiters in the past 1.5 weeks, which wasn’t the case in the last few months! Most are from startups and small/mid-sized companies”

Senior engineer, 7 YOE, Switzerland:

“I am seeing an uptick in inbound messages from recruiters. Until recently, it was one per week or less. Around a third are from crypto startups, the rest are mid-to-large scaleups. None are from Big Tech.”

Staff engineer, scaleup, 10 YOE, the Netherlands:

“I am seeing an increase in reach outs in the last 2 months. This is mostly from larger companies, and some from startups.”

Software engineer, 5 YOE, the Netherlands:

“I have recently set my profile as ‘open to work.’ Although I don't have any big names in my profile and I switched technologies, I still get on average an inbound request from a recruiter every day, even weekends!

Data engineer, 3 YOE, France:

“I'm receiving some more reach outs recently, but not that much compared to previous months. The difference is in the quality of the offers: scaleup reach outs are back, but not as intense as 3 years ago.”

Based on readers’ feedback, it seems that something changed in several countries’ recruitment markets around 2 months ago:

Backend engineer, 15 YOE, Germany:

“In the spring, I was let go from my company, together with my whole team. As I was browsing job portals, I saw surprisingly few software engineer job postings, but a lot of DevOps openings. So I started to apply for DevOps positions instead.

Fast forward to June, and software engineer listings were back in larger numbers. Since then, I managed to land a backend engineer position, and I’m starting my new job in September.”

Frontend engineer at a full-remote tech startup, France:

“Definitely seeing a lot more recruiter interest for roles that are actually not bad. It had died down last year, but it's been picking up for 2-3 months now.”

Software test architect, 15 YOE, the Netherlands:

“I confirm that I started to get messages from recruiters in the last 1-2 months.”

Engineering director, 10 YOE, publicly traded tech company, Germany:

“My anecdotal evidence is we’re seeing an increase in the last 2 months. In my inbox there is a healthy mix of messages from start-ups/scale-ups that have closed a funding round, established companies, and headhunters trying to fill confidential roles. There is a clear uptick of messages related to startups with fresh funding.”

Data engineer, 4 YOE, Germany:

“Definitely something is moving, received 10 messages in the last 20 days for data engineer roles in startups (in Berlin, or remote in Germany). This is around a 10x increase: I got only 2 messages in previous 3 months (April to June.)”

Hiring for principal, staff and tech lead roles is new to some readers. Two engineers mentioned they were surprised by reach outs for these roles, which was not the case even in 2022! A principal software engineer at a publicly traded company with 15 YOE, said:

“Startups and scaleups started to reach out for principal/staff engineering and tech lead roles seemingly out of the blue, a month or two ago. I get around one reachout per week, after months of nothing. The interesting part is that in 2020-2023 my title was the same, but I only got senior engineer roles as inbound, not principal ones like now.”

Despite the uptick, it’s still hard to get an interview. A director of engineering at a publicly traded tech company in the Netherlands, shared:

“I've definitely seen an uptick in recruitment messages, as well as callbacks from recruiters.

Even from senior people, I hear that even for very strong job matches they don't even get a call back. No first call, nothing. Just straight up rejection. But this also seems to have improved with a lot more companies being open to at least the first chat.

One common feedback I hear from friends and colleagues, even senior people or from FAANG, is that even the big ones are hiring too (Uber, Booking, Meta, Adyen, etc.) Still, you get an instant rejection, not even a first call; when a couple of years ago you would always get at least the first call.”

Southern Europe

VC-funded and pre-IPO companies seem to have hastened recruitment in the last two months:

Security engineer, scaleup, 15 YOE, Spain:

“I can confirm that in the last two months I have received more messages from companies and recruiters offering a job, compared to the last 2 years! All the companies are VC backed, and Series A, or pre-IPO stage. So it looks like there is an interest in growth again.”

Senior engineering manager at a US scaleup, 20 YOE, Italy:

“I see an uptick from Series B/A startups and local companies (many hybrid, a few full remote, and one or two office based). A few come from public tech companies. Unfortunately, a few are recruiters just collecting CVs.

All in all, it seems like the market is back to pre-Covid days.”

Tech lead, scaleup, 10 YOE, Spain

“I can confirm – I'm not sure why! – there's been a spike in the offers received during the last 3-4 weeks. Most are startups.”

Parts of southern Europe could be seeing more interest due to the lower cost of employing software professionals in the region. A tech lead at a scaleup in Portugal notes that several companies are moving their engineering to the country, or expanding current teams there:

“Last week, I got 10 (!!) inbound recruiter messages in just a day. Usually, I get this many over a week or more. Something is happening.

50% of messages are from startups. The rest is a mix of mid-size companies and established big companies (the smallest portion.) As context, in Portugal a big portion of companies recruiting are moving here. Either starting to build their IT hubs or reinforcing the existing team.”

For context, Portugal is especially popular in tech recruitment as a gateway to hire and relocate software professionals from Brazil – the world’s 7th largest country by population, with 215M people.

3. Why an uptick now?

Read more

I wrote for Linux Journal from 1996 to 2019, the final years as editor-in-chief. After ownership changed and the whole staff turned over, the new owner, Slashdot Media, agreed to keep the server up so nothing would be 404’d. I am grateful that they have kept that promise. I should add, however, that some of the archive seems to be missing—or so I assume because keyword searches on Google, Bing, and the site itself fail to bring up some items. Fortunately, I have an archive of my own writing for the magazine—or at least of the final drafts I submitted. Since the cadence of this blog has fallen off a bit, I think a good way to fill open spaces in time is to re-publish columns I wrote for Linux Journal when Linux was still an underpenguin and the open source movement was still new, growing, and a threat to the likes of Microsoft. (Which has since flipped its stance. We’re well past GandiCon 4 now.*) This piece is one example: a small hunk of history that bears re-telling. (And forgive the rotted links, because, alas, the Web is a whiteboard.)

Linux For Suits

August 2001

Setting the terms

Back in May, Craig Mundie, Senior Vice President with Microsoft, gave a speech at NYU’s Stern School of Business that announced the terms by which Microsoft was cracking open — barely — its source code. He called the company’s new licensing model “shared source.” (I just wrote “scared source” by mistake, which tells you where my mind is going.)

The fact that Microsoft would start rapping about any kind of source code, and modify it with a fresh new euphemism — shared — caused immediate tissue rejection in the hacker cultural body. Leading hackers so certain of their own Truth that they refuse to appear on each other’s t-shirts were suddenly gathered around their collective keyboards to craft a single response that would say, in polite terms, “Embrace and extend this, dude.”

The result was an open letter published on Bruce Perens’ site (Perens.com), and signed by Bruce and a quotariat of free software and open source luminaries: Richard Stallman, Eric S. Raymond, Guido Van Rossum, Tim O’Reilly, Larry Augustin, Bob Young, Larry Wall, Miguel de Icaza and Linus Torvalds. (An anonymous coward on Slashdot wrote, “It’s like a human Beowulf cluster!”) While critical and challenging to Microsoft, its bottom line was inviting:

We urge Microsoft to go the rest of the way in embracing the Open Source software development paradigm. Stop asking for one-way sharing, and accept the responsibility to share and share alike that comes with the benefits of Open Source. Acknowledge that it is compatible with business.

Free Software is a great way to build a common foundation of software that encourages innovation and fair competition. Microsoft, it’s time for you to join us.

Mundie responded with a piece in CNET that framed his argument in terms of economics, manufacture and the PC’s popularity:

… this is more than just an academic debate. The commercial software industry is a significant driver of our global economy. It employs 1.35 million people and produces $175 billion in worldwide revenues annually (sources: BSA, IDC).

The business model for commercial software has a proven track record and is a key engine of economic growth for many countries. It has boosted productivity and efficiency in almost every sector of the economy, as businesses and individuals have enjoyed the wealth of tools, information and other activities made possible in the PC era.

Then he took on the GPL — the Free Software Foundation’s General Public License:

In my speech, I did not question the right of the open-source software model to compete in the marketplace. The issue at hand is choice; companies and individuals should be able to choose either model, and we support this right. I did call out what I believe is a real problem in the licensing model that many open-source software products employ: the General Public License.

The GPL turns our existing concepts of intellectual property rights on their heads. Some of the tension I see between the GPL and strong business models is by design, and some of it is caused simply because there remains a high level of legal uncertainty around the GPL– uncertainty that translates into business risk.

In my opinion, the GPL is intended to build a strong software community at the expense of a strong commercial software business model. That’s why Linus Torvalds said last week that “Linux is never really going to be a rich sell.”

This isn’t to say that some companies won’t find a business plan that can make money releasing products under the GPL. We have yet to see such companies emerge, but perhaps some will.

He added,

What is at issue with the GPL? In a nutshell, it debases the currency of the ideas and labor that transform great ideas into great products.

It would be easy to dismiss all this as provocation in the voice of boilerplate — or worse, as what one überhacker called “a declaration of war on our culture.” But neither of those responses are useful to folks caught in the middle — the IT professionals this column calls “suits.”

As it happened Eric Raymond and I were both guests on the May 14 broadcast of The Linux Show. When conversation came around to the reasoning behind open source rhetoric, Eric said, “We used the term open source not to piss off the FSF folks, but to claim a semantic space where we could talk about issues without scaring away the people whose beliefs we wanted to change.”

This has been an extremely successful strategy. Even if IT folks don’t agree about what “open source” means, it’s still a popular topic. Everybody who talks about open source inhabits its semantic space. But conversing is not believing. Remember Eric’s last seven words. These are still people whose beliefs we want to change.

Changing other people’s beliefs isn’t like changing your shoes. It’s like changing other people’s shoes. There’s a lot of convincing to do. Even if the other guy’s shoes are ugly and uncomfortable, at least they’re familiar. And in this case, familiar doesn’t cover it. In the IT world, Microsoft platforms, software and tools are the prevailing environment. Of course, we used to say the same thing about IBM. Things do change.

Did anything change when Craig Mundie tried to embrace and extend the conversation about source code? I think so. Mundie’s response to Bruce’s letter looked like a poker move to me. He said, “We’ll see your source and raise you one shared“. What was our response? As a unit, nothing. the Beowulf Cluster broke up over the usual disagreements. It wasn’t pretty.

Perhaps it’s just as well. Two of the original signers told me they felt the letter was skewed “from the pragmatic to the ideological.” If so, Mundie read the letter well, because the ideology is exactly what his response attacked.

When this kind of thing happens, is the right choice to attack back?

We need to be careful here. When Microsoft decided to release a free browser, Marc Andreessen said “In a fight between a bear and an alligator, what determines the victor is the terrain. What Microsoft just did was move into our terrain.” He also called the operating system “just a device driver.” Today Microsoft is wearing alligator shoes.

And now the bear sits in our semantic space, talking business trash.

You might recall what I wrote exactly a year ago, when every mainstream publication was running stories about how the feds were going to break up Microsoft. I said Microsoft was going to beat the rap by baiting Judge Jackson, giving the company a winnable case in Appeals Court that it lacked in Judge Jackson’s.

So: are they baiting us here? You betcha. But the appeals court in this case is the whole IT community that remains Microsoft’s customer base.

What they want us to do is defend the incomprehensible: namely, all the stuff we can’t stop arguing about.

The tactical picture becomes clear when you look at this diagram from the Free Software Foundation’s philosophy page. From the perspective of both Microsoft and its customers, the one thing that’s easy to understand is in the upper right. By aiming insults at the GPL in the lower left, they rally everybody in the Free/Open communities to defend what those outside those communities have the most trouble understanding: the FSF’s belief that owning software is a Bad Thing.

If we defend that position or respond by using “proprietary” as an epithet, we’ll lose. “It takes two to tango,” one commercial developer wrote to me in the midst of all this. “If Microsoft ever chose an enemy who was willing to share the cursor, they would finally have met their match. It hasn’t happened yet. Your friends are still saying ‘It’s All About Us’ which is complete bullshit. They never talk about anyone but Microsoft and themselves.”

Which is why we need to embrace and extend the most precious thing that Microsoft has and we don’t, which is customers. Why? Because it’s their hearts and minds we need to win, and Microsoft is busy ignoring them.

Look back at Mundie’s text. Note that his point of view is located not with customers, but with the commercial software industry. Big difference. Yes, commercial software companies do face a choice among many different business models and licensing schemes, including the GPL. But this argument isn’t just about business models. It is not the exclusive concern of Supply. It’s also about Demand. In some ways the real argument isn’t between Supply and Supply: one “shared” and one “open”. It’s between Supply and Demand.

On the demand side, customers are using software of many different types, from many different sources. Whether they know it or not, most large enterprises are already full of applications and development tools from the free software and open source communities. Are they choosing those “solutions” just because they like one party’s source license or another party’s business model? No. They’re using it because it’s available and practical.

We need to relocate our concerns to the demand side of markets. What is it that works, and why? Specifically, what “solutions” work for everybody? Our best example, our ace in the hole, isn’t Linux alone. It’s the Net and all the free and open software that accounts for its ubiquity. Our communities created and proliferated that software because we know something about its nature that the Microsofts of the world do not.

The “licensing structures” Microsoft cares most about all rely on conceiving code as capital: as a manufactured good. Concepts like “intellectual property” are easy to understand and argue about as long as one continues to conceive of that property in material terms. But code is not material, and no amount of lawmaking or marketing can make it material.

The deepest fact in this matter is not that software wants to be free, but that code wants to be public. Meanwhile, too many of our laws and business practices cannot comprehend this simple fact. It’s too far below their immediate concerns. Like the core of the Earth, it’s nice to have but too deep to appreciate.

The problem for Microsoft is that it lives in a world increasingly built with public code that oozes like lava out of the free and open ground below everybody. We’re making a whole new world here, but we’re doing it together. And that includes Microsoft, which in fact does contribute to common infrastructure. SOAP is a good example.

Indeed, much good has been produced by what Mundie calls the “commercial software model” or it wouldn’t have customers. But the Net on which all of business increasingly relies is not a product of that model, even though business is certainly involved.

What free/open hackers know and Craig Mundie doesn’t (yet), is that there is much to the nature of code that can neither be comprehended nor represented by the conceptual system Mundie employs — not because it’s insufficient in scope but rather because it’s operating only at the commercial level. At a deeper level — the nature of software itself — the principles of business don’t apply, for the same reason that the principles of mechanics don’t apply to chemistry, even while mechanics depends on chemistry as a deeper principle. You’d rather make a clock out of iron than sodium.

Business needs more of that public infrastructure. It needs programs, operating systems, device drivers, file formats and protocols that everybody can use because nobody owns them. So do commercial developers and their customers: all of Supply and all of Demand. But those terms are too abstract. Think of the entire mess as a bazaar.

If we embrace the whole software bazaar, we can open hearts and extend minds. If we refuse to share the cursor, we cease to represent the bazaar. At that point the choice is between a cathedral and a cult.

Links:

Mundie’s speech: <http://www.microsoft.com/presspass/exec/craig/05-03sharedsource.asp>

Perens’ response: <http://www.perens.com/Articles/StandTogether.html>

Mundie”s response: <http://www.internetnews.com/intl-news/article/0,,6_766141,00.html>

*That happened in 2016.

This post implements a basic ASP.NET Core API using .NET 9 and the Microsoft OpenAPI implementation. The OpenAPI Nuget package supports both Controller based APIs and minimal APIs. Until now, we used excellent solutions like NSwag to produce the API schemas which can be used to auto-generate client code.

Code: https://github.com/damienbod/WebApiOpenApi

Setup

A .NET 9 project can be created using the .NET Web API templates. The required SDKs or Visual Studio version needs to be installed. The OpenAPI Json definitions can be created using the Microsoft.AspNetCore.OpenApi Nuget package. Microsoft learn docs have all the different possibilities for implementing this depending on your preferred development environment. Here’s an example using powershell:

Install-Package Microsoft.AspNetCore.OpenApi -IncludePrerelease

JWT OAuth authorization is used in this example for the API. I installed the following packages:

Microsoft.AspNetCore.OpenApi Microsoft.AspNetCore.Authentication.JwtBearer Microsoft.Extensions.ApiDescription.Server

The OpenAPI services are added to the project. The API uses JWT Bearer and OAuth and so the OpenAPI definitions should also add this. An BearerSecuritySchemeTransformer implementation was created like in the Microsoft documentation.

builder.Services.AddOpenApi(options => { options.AddDocumentTransformer<BearerSecuritySchemeTransformer>(); });

The BearerSecuritySchemeTransformer class implements the OpenAPI specific definitions. If using cookies, only the document.Info is required and this can be added directly in the services without an extra class. From the Microsoft docs:

internal sealed class BearerSecuritySchemeTransformer(IAuthenticationSchemeProvider authenticationSchemeProvider) : IOpenApiDocumentTransformer { public async Task TransformAsync(OpenApiDocument document, OpenApiDocumentTransformerContext context, CancellationToken cancellationToken) { var authenticationSchemes = await authenticationSchemeProvider.GetAllSchemesAsync(); if (authenticationSchemes.Any(authScheme => authScheme.Name == "Bearer")) { var requirements = new Dictionary<string, OpenApiSecurityScheme> { ["Bearer"] = new OpenApiSecurityScheme { Type = SecuritySchemeType.Http, Scheme = "bearer", // "bearer" refers to the header name here In = ParameterLocation.Header, BearerFormat = "Json Web Token" } }; document.Components ??= new OpenApiComponents(); document.Components.SecuritySchemes = requirements; } document.Info = new() { Title = "My API Bearer scheme", Version = "v1", Description = "API for Damien" }; } }

The middleware needs to be added to the pipeline. You should only add the OpenAPI in development mode unless you require this in production.

app.MapOpenApi("/openapi/v1/openapi.json");

ASP.NET Core supports two types of API implementations; Controller based and minimal APIs. Controller based APIs are used in this project. This API endpoint requires a valid access token and have the different Endpoint definitions.

[Authorize(AuthenticationSchemes = "Bearer")] [ApiController] [Route("[controller]")] public class WeatherForecastController( ILogger<WeatherForecastController> _logger) : ControllerBase { [EndpointSummary("This is a summary from OpenApi attributes.")] [EndpointDescription("This is a description from OpenApi attributes.")] [Produces(typeof(IEnumerable<WeatherForecast>))] [HttpGet("GetWeatherForecast")] public IActionResult Get() { _logger.LogDebug("GetWeatherForecast with OpenAPI definitions"); return Ok(Enumerable.Range(1, 5).Select(index => new WeatherForecast { Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)), TemperatureC = Random.Shared.Next(-20, 55), Summary = Summaries[Random.Shared.Next(Summaries.Length)] }) .ToArray()); }

You can also add definitions for POSTs and parameters or whatever you require.

[EndpointSummary("This is a second summary from OpenApi attributes.")] [EndpointDescription("This is a second description from OpenApi attributes.")] [Produces(typeof(IEnumerable<WeatherForecast>))] [HttpPost("PostWeatherForecast")] public IActionResult PostWeatherForecast( [Description("parameter post item using OpenApi")] WeatherForecast weatherForecast) { return Ok(weatherForecast); }

When the application is started, the Json OpenAPI definitions can be downloaded and used.

Notes

I am not sure how good the OpenAPI specifications are and need to validate how to define the different status codes in the specifications with the correct return types.

Links

https://github.com/martincostello/dotnet-minimal-api-integration-testing

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/minimal-apis/aspnetcore-openapi

https://learn.microsoft.com/en-us/aspnet/core/web-api/action-return-types

https://github.com/RicoSuter/NSwag

A ChatGPT generated image

I decided to do some retrieval augmented generation(RAG) with Llama 3.1 8B to see how it went. In short, it went pretty well with my limited testing. More interesting is probably how I set it up to perform the RAG.

First, I didn’t want to reinvent the wheel. Not yet. I was looing for a way to provide RAG with Llama 3.1 without coding up a user interface for embedding multiple documents and creating chat bot that would use those embeddings. The combination I created was AnythingLLM and LM Studio backed by Llama 3.1 8B Instruct q_8 model.

I ran LM Studio as a server using Llama 3.1 underneath. Ran AnythingLLM as a front end to the AnythingLLM embedder and the AnythingLLM provided LanceDB vector store with the combined set pointing to LM Studio server as the LLM provider.

For AnythingLLM you can then create workspaces that allow for document embedding for that workspace into the vector store. I tried out a doc I was especially curious to chat across as it was both a large PDF and a topic of political and policy interest — the 900+ page Project 2025 “manifesto”. It took 3 minutes for it to create the embedding. I am not sure how many dimensions it embeds but AnythingLLM will show you how many vectors it creates. In this case 5038 which is smaller that I would have expected for 900 pages. It does provide some basic settings around embedding; you can set the chunk size as up to 1000 characters and the overlap (in characters). I used the default which was the maximum of 1000 character chunk size and an overlap of 20. At first I though the overlap was in percentage but it states it is in characters, not tokens or percentage. I am suspicious that 20 character overlap is really enough for the best context and may play with that. That seems a really small number to me; I know Microsoft on their embedding guidance recommends 10% to start with.

You can also select your embedding model. I have so far only tried the AnythingLLM Embedder but I will likely try other local embedding models and probably Azure OpenAI as I have access to that via my Azure account.

After setting up my environment I took a workspace I had created and uploaded the Project 2025 document and began to query. My first query was about IVF and it reported back that there was no mention of IVF but did provide a very general description of the the document as a whole. So I decided to drill in a little more deeply and asked about rights of embryo’s and fetuses. It came back with what seemed to be a response I would have expected from the document. That’s when I realized one issue with my test. I hadn’t read the document so I didn’t really know if it provided an accurate analysis. Based on the results it seemed to likely be accurate but I couldn’t be certain. In the image below you can’t see my initial prompt but it was “You are a society and government policy analyst. What does the 2025 Mandate For Leadership say about IVF?”

This lead me to a different test.

I created a new workspace and added a document that I was familiar enough with. I could have added it to the existing workspace but wanted to keep a separation for testing purposes. The document uploaded was 43 pages. I didn’t notice how long it took to create the vectors but it did end up with 177 as the vector account. Again, lower than I would have expected.

I asked it what should have been a pretty softball question about the 10 principles of SSI (self-sovereign identity). The embedded document specifically summarizes them. Keep in mind that it could use the Llama model and/or the document provided. I didn’t restrict/suggest it to using just the document that had been embedded in the vector database. It came back again with what seemed to be a valid response. What I now need to do is compare the document to the inferences and see if there was hallucination or how much it may or may not have mixed the RAG with the underlying Llama model values. It does provide a citation and listed the document, but I don’t know if that means all the inference came from the document or if maybe just some of the inference came from the cited work. I would expect it to mean the latter and thus 1–100% of what it responded was from the document. I may never know the answer to that question. I do know when I created yet another workspace with no document and provided the same prompt it did come back with some responses that I knew were not quite accurate. The inference when it had the RAG was definitely more accurate.

The “thinking” and the returned tokens/sec were slow but this is running on a local machine with very limited horsepower. (If you wish to see the machine specs check out a prior post “Started Llama 3.1 8B Locally”). If I had an OpenAI API account I’d likely try it out against that LLM endpoint with the RAG and see how it performed, but I currently don’t. Perhaps I can do that in Azure as well.

Keep in mind I had to upload the documents for RAG individually. I don’t see an option to point it to a folder and let it walk the folder recursively. AnythingLLM does say on the upload screen that it “supports text files, csv’s, spreadsheets, audio files, and more!” I’ll have to test it out on some of those items. You can also add website URLs and it has a data connector tab for some other inputs — see the below image.

All in all, it behaved as I would have hoped for the environment. It is after all only an 8 billion parameter LLM model. AnythingLLM gave me a lot of flexibility with very little effort and turned my LM Studio server into something more than just the underlying Llama 3.1.

Oh I didn’t mention AnythingLLM also has some nature of agent support. I’ll certainly be trying that out in the future.

I can’t believe it’s been over two years since I last published an article here! Time really flies.

The crazy part is that over these last two years, I’ve done more writing than ever; it just hasn’t been public.

Several years back, I started a journaling habit, and since then, most of my writing energy has been redirected from my website to my private journal.

While writing in private feels liberating, as I can dump every little thought in my head into an archive, I find myself missing the more structured and thought-out articles I used to push myself to publish.

There’s something magical about writing down your thoughts and sharing them with the world, even if nobody sees them. It forces you to rethink your position, clarify your thoughts, and distill the mess in your head into something direct and actionable.

Over the last couple of years, I have missed that feeling, and the itch to continue writing has returned.

All this is simply to say that I’m writing again. The title of my website has always been “Random Thoughts of a Happy Programmer,” so please stay tuned for more of my innermost thoughts and experiments.

Orie Steele and I have updated the “Fully-Specified Algorithms for JOSE and COSE” specification to incorporate feedback from IETF 120 in Vancouver. Specifically, the registrations for fully-specified Elliptic Curve Diffie-Hellman (ECDH) algorithms in draft 03 were removed, along with the previously proposed fully-specified ECDH algorithm identifiers, while continuing to describe how to create fully-specified ECDH algorithms in the future, if needed.

The specification is available at:

https://www.ietf.org/archive/id/draft-ietf-jose-fully-specified-algorithms-04.html

I started a load of Llama 3.1 8B onto my local machine. It’s not the beefiest model but is the model that I could support on my aging machine and GPU. I have not yet had time to play with it but this is step one of a multi step experiment.

It took me less than 30 minutes to get up and running and that is an estimate on the far end. The bulk of that time was deciding if a specific tool was supported on my machine. I needed to make sure my processor had AVX2 support. Once I got over that hurdle it was easy peasy. I decided not to start with Ollama simply because LM Studio intrigued me, and I liked the interface for adjusting parameters right in the interface plus the more pleasing interaction experience than just the command line. LM Studio allowed me to install in minutes, select a chat interface, or server, and then provided the interface which allowed me to select which of the models I have installed and I was off and running for a chat session to begin with. The response time was good and returned output at about as fast as I could read it.

Why the 8B model? Simply because I don’t have the machine specs to run the 70B. You really need multiple terabytes of SSD storage, a much better GPU, and twice as much RAM at a minimum. The 8B is more than sufficient for me to start testing agent creation and execution.

My Machine Specs For the 8B

As an example of the machine specs I was able to get the 8B running under with very reasonable performance.

i7–107000k processor at 3.8 GHz 16 GB RAM Nvidia GTX 1060 with 6GB of RAM

I have a MacBook as well but it is a 2019 MacBook with an Intel processor. Llama 3.1 only supports M1+ processors.

70B Machine Specs

According to the 8B model, when asked, the 70B model needs:

Processor: i7 or higher with 8 cores and 16 threads.
- So not much required in the CPU department. GPU: A high-end NVIDIA GeForce or Quadro GPU with at least 24 GB of VRAM (e.g., GeForce RTX 3080 Ti or Quadro RTX 8000). Memory: At least 64 GB of RAM, preferably 128 GB or more.
– This will ensure that you have sufficient memory to handle the model’s massive parameter count (70 billion parameters) and other system resources.
- Not sure I’ll ever have machine with 128 GB of RAM A high-speed storage drive with a large capacity (at least 1 TB)

Here’s the latest installment in the series on working with LLMS: https://thenewstack.io/choosing-when-to-use-or-not-use-llms-as-a-developer/

For certain things, the LLM is a clear win. If I’m looking at an invalid blob of JSON that won’t even parse, there’s no reason to avoid augmentation. My brain isn’t a fuzzy parser — I’m just not wired to see that kind of problem, and that isn’t likely to change with effort and practice. But if there are structural problems with code, I need to think about them before reaching for assistance.

The rest of the series:

1 When the rubber duck talks back

2 Radical just-in-time learning

3 Why LLM-assisted table transformation is a big deal

4 Using LLM-Assisted Coding to Write a Custom Template Function

5 Elevating the Conversation with LLM Assistants

6 How Large Language Models Assisted a Website Makeover

7 Should LLMs Write Marketing Copy?

8 Test-Driven Development with LLMs: Never Trust, Always Verify

9 Learning While Coding: How LLMs Teach You Implicitly

10 How LLMs Helped Me Build an ODBC Plugin for Steampipe

11 How to Use LLMs for Dynamic Documentation

12 Let’s talk: conversational software development

13 Using LLMs to Improve SQL Queries

14 Puzzling over the Postgres Query Planner with LLMs

15 7 Guiding Principles for Working with LLMs

16 Learn by Doing: How LLMs Should Reshape Education

17 How to Learn Unfamiliar Software Tools with ChatGPT

18 Creating a GPT Assistant That Writes Pipeline Tests

19 Using AI to Improve Bad Business Writing

20 Code in Context: How AI Can Help Improve Our Documentation

21 The Future of SQL: Conversational Hands-on Problem Solving

22 Pairing With AI: A Senior Developer’s Journey Building a Plugin

23 How LLMs Can Unite Analog Event Promotion and Digital Calendars

24 Using LLMs to Help Write a Postgres Function

25 Human Insight + LLM Grunt Work = Creative Publishing Solution

When explaining the W3C Verifiable Credential Data Model (VC), Open Badges 3.0, and LERs, I’m often asked about the role of blockchain and if I’m not asked, I feel compelled to explain because misconceptions and confusion are common. Blockchain is one of those technologies that is at once fascinating and cringey, useful and over-used. The business models are still experimental and when it comes down to it, there are simpler, more efficient, and more affordable methods for issuing data rich portable, verifiable digital credentials that meet the needs of issuers, give learners and earners easy access to their data, and a private way to prove that their digitally-signed achievements are theirs for as long as they need them.

You could say that I’ve spent some time thinking about this. In 2015, I co-founded a group called BadgeChain with other Open Badges community members that explored how Open Badges could be more verifiable and have clearer provenance if stored on blockchain. This coincided with Philipp Schmidt and Kim Hamilton Duffy’s leadership on Blockcerts, an open specification for issuing Open Badges on Bitcoin and Ethereum. Learning Machine continued with this work, which was later acquired by Hyland. In 2020 I was the lead researcher on the Education and Blockchain initiative with the American Council on Education and the US Department of Education where we reported on the potential uses of blockchain and then supported several projects that used blockchain or blockchain-like technology for education and training credentials.

I truly believed that we were creating the future and would help society by leveraging what seemed like an obvious use of distributed digital ledgers but what we all learned from this was:

Blockchains are expensive and energy-sucky ways to store and verify credentials It’s a headache to explain to and request public keys from credential recipients Storing data on a blockchain (especially long-living ones like Bitcoin and Ethereum), even when hashed, is risky for PII (Personal Identifiable Information) and means that once it’s on a chain, it can’t be changed or removed. So, if any mistakes are made or data has changed, there’s no way to remove it. It can create a vendor lock for the issuer and the recipient. When the vendor/network goes away, where do those credentials go? It doesn’t free data trapped in centralized silos. It creates new types of silos.

During this time and since, the Credentials Community Group at W3C (CCG) started developing the W3C Verifiable Credentials Data Model (VC). The use cases for this were originally sourced from the members of Open Badges community in 2017. The W3C Verifiable Credentials for Education Task Force was founded soon after by the Digital Credentials Consortium (DCC) and it was this task force which agreed to propose Open Badges 3.0 to 1EdTech in 2021. Now Open Badges 3.0 is final and W3C VC 2.0 is almost complete.

Issuing Open Badges 3.0 is not only about how the achievement is described or the structure of the metadata. The metadata is a critical component to the standard but the bigger upgrade is the alignment to W3C VCs so that digital education credentials like badges, micro-credentials, certifications, and even degrees could be stored, shared, verified, and used on the web in the same secure way as driver’s licenses and citizenship documents that are also issued as W3C VCs. The intention is for all of these credentials to be understandable and verified similarly so that they can be used together.

W3C VCs are cryptographically verifiable, meaning that verfiers can check via software that the data hasn’t changed since it’s been signed, validate the identity of the issuer, and confirm that the credential hasn’t expired or been revoked. They are digitally signed by the issuer and provided to the credential earner. From that point on, that credential belongs to the earner and whatever they do with it is their business only. It may be that issuers ask if earners are willing to tell them where they use their credentials, but inherently, the basic premise of VCs is that they can be used independently. Just because the data is sourced from the issuer does not mean how that data is used should be shared with the issuer if the earners don’t consent. Would you want the DMV to know every time you share your driver’s license? Should the DMV get paid if you use your driver’s license? Even if it was digital? Why should it be any different for education and training credentials?

Admittedly, there are interesting blockchain models to explore that are adjacent to or support W3C VCs and Open Badges 3.0. For example, we’re just starting to touch upon the usefulness of issuer registries and blockchain could be useful there; blockchain-like technologies like Fluree are fueling systems like the Trusted Learner Network (TLN) where distributed nodes of data can be accessed by students and alumni to produce VCs; systems like Ceramic and IPFS can be used for longevity and distribution of evidence and supporting files; some Decentralized identifier methods that use blockchain are being tested around the world.

A model where verifiers pay to access and verify credentials in a tokenized blockchain system compromises the privacy of the credential recipients and traps their credentials in a closed network. In the end, it’s the credential earners that get forced into a closed system where others could profit from their credentials that represent the investment of their time, work, and money. Also, what’s the incentive for verifiers to pay tokens for access to credentials when they could evaluate and accept other credentials on the web for free?

That is the opposite of the intention of W3C Verifiable Credentials and Open Badges 3.0. At the DCC along with many others, we promote the use of W3C VCs as distinct entities of digitally signed achievements that are in the hands of those who earn them, who choose how to use them, and who or what can have access to them. We need to make it easier and affordable for verifiers to get onboard and understand how VCs and Open Badges work so that they become commonplace and useful on the web for everyone.

For more information on the DCC, including membership, projects, and events, visit our website, wiki, and Linkedin. To receive our newsletter and invitations to webinars and events sign up for our community mailing list.

Do We Need Blockchain for LERs? No. was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Digital Credentials Consortium (DCC) is advancing the use and understanding of portable, verifiable digital credentials in higher education through open source technology development and leadership, research, and advocacy. What are portable, verifiable digital credentials and why is it important that we advance their use?

Digital credentials at their most basic definition are electronic assets that describe an achievement or a qualification. Credential Engine’s Credential Transparency Description Language Handbook describes a credential as “A qualification, achievement, personal or organizational quality, or aspect of an identity typically used to indicate suitability”. There are many different credential types that can be used in education, training, and industry as learning and employment records (LERs) to describe achievements and provide details on related skills and competencies.

When referring to credentials as “portable” and “verifiable” we are indicating that they have been created in alignment with the key web standards that are used to issue digitally signed, metadata rich digital credentials. The W3C Verifiable Credentials (VC) standard is the web standard for digital signatures and proving claims made on the web. Open Badges 3.0 is a standard conformant with W3C Verifiable Credentials that is specifically tailored to issue education and experience digital credentials. Both standards result in distinct electronic assets that can be held by individuals in wallets or other storage to be used later.

Through its participation in community standards organizations including the W3C and 1Edtech, the DCC has helped to develop these standards and is advancing their use and understanding so that individuals may have more access to their data and have the agency to freely and privately share and use their credentials. To accomplish this, the DCC is advocating that issuers of digital credentials use software and vendors that prioritize the use of the LER VC standards. Our members are issuing micro-credentials and even degrees as Open Badges and W3C Verifiable Credentials so that their students and alumni can have digitally signed, data rich markers of achievement that describe what they know and can do that are usable anywhere on the web (and in the world) for their lifetimes.

For more information on the DCC, including membership, projects, and events, visit our website, wiki, and Linkedin. To receive our newsletter and invitations to webinars and events sign up for our community mailing list.

What Are Portable, Verifiable Digital Credentials? was originally published in Digital Credentials Consortium on Medium, where people are continuing the conversation by highlighting and responding to this story.

AI generated crew image. Note that some folks are backwards. :-)

This weekend’s AI adventures was into agents, crewAI specifically. I learned a lot, and it made me even more eager to dig deeper.

One of the YouTube videos I watched was a line-by-line Python walkthrough of a crewAI example. I think the example was taken from the official docs/examples but the line-by-line presentation was invaluable. It was amazingly and relatively easy to set up a three task and three agent process along with some tools for it to run with. I was impressed. I plan to play around with the task and agent configurations to see the output. The one downside is that using OpenAI GPT4 may become prohibitively expensive in making lots of experimental calls, plus the cost of crewAI itself.

I don’t yet have a feel for the cost for such usage and each run may vary not only due to configuration but because the agents might run more iterations one time than they do another. It depends on when they think they are done. I have seen that you can run Llama locally and even videos on how to connect crewAI to local instances of Llama but I am not sure that will meet my experimentation needs. I have to first play around with Llama to know that.

Another consideration is if I do this on my PC with it’s 16GB of RAM and measly Nvidia GTX 1060 card or do it on my Macbook with 32 GB of RAM but I am not sure what the GPU specs are. Some more research is in order and perhaps side by side comparisons.

I’ll report back on my general findings.

If you don’t live in a swing state, but would like to do more than just send money to help encourage voter turnout in those places, what are your options? For me the best one is Vote Forward, which orchestrates letter-writing to registered voters. I sent hundreds of such letters in 2020 and am aiming to do lots more, with help from friends, this time around.

Even if I lived in a swing state, I’m not someone who’d be comfortable knocking on doors. And the last thing I want to do is pester people in those places with yet another unwanted phone call or text message. The Vote Forward method is perfect for me personally, and I also think it’s the most clever and sensible way to encourage voters in other states. Here’s how it works.

You “adopt” voters in batches of 5 or 20. I just adopted my first 100: 20 in each of Ohio, Pennsylvania, Michigan, New Hampshire, and North Carolina. You download each batch as a PDF that prints 21 pages. Page one has the instructions and the list of registered voters’ names and addresses

The fact that you write the letters (and address the envelopes) by hand is a great idea. We receive very few hand-addressed letters nowadays, I think they have a pretty good chance of being opened. And once opened, the hand-written message is again unusual. The fact that somebody made the effort to do that signals a rare kind of authenticity.

Likewise, I think the nonpartisan tone of the message is unusual and conveys authenticity. I wish voting were mandatory in the US, as it is in Australia and elsewhere. However the chips fall in November, I would like to know that the result truly reflects what everyone thinks. My message last time was something like:

“… because it’s not really a democracy unless everyone’s voice is heard.”

Pages 2-21 are the letter templates. They look like this:

The hardest part for me was the handwriting. I famously struggled with cursive writing in fifth grade. By the time I reached high school I had reverted to printing. Then, in college, I realized that cursive is more efficient and relearned how to do it. I had to relearn all over again in 2020 because cursive was the fastest way to write all those letters. And I’ll probably have to relearn again this time. I suspect many in younger generations never learned cursive at all, in which case writing the letters by hand will be even harder. So: keep the message short!

If you’ve received a link to this post directly from me, it’ll come with an invitation to drop by our house, hang out on the porch, and help me complete batches of these letters. Otherwise, I hope you might try this method yourself, and/or share it with others. In the past week I’ve switched from doomscrolling to hopescrolling and that’s a huge relief. But I also want to do something tangible (again, beyond donations) and this will be my focus. It feels good to do the work, and will feel really good when I visit the post office sometime in October and drop off a big stack of hand-addressed envelopes.

But is it effective? That’s another thing I like about Vote Forward. They’ve made a sincere effort to measure the impact. And they are honest about the findings: the measurable effect is small. I’ll give them the last word here.

Why should we get excited about small differences?

Because getting people who don’t vote to show up at the polls (or mail in a ballot) is actually pretty hard. Most of the factors that affect whether people vote are tied to big, structural issues (like voter ID laws or polling place accessibility) or deep-seated attitudes (e.g., a lack of faith that elections matter). Given these obstacles, boosting turnout by even a small amount is a real achievement! And, when it comes to politics, we know that many races are decided by tight margins, so a small boost in turnout can translate into a meaningful difference in electoral outcomes.

The OpenID Foundation has approved the Fourth Implementer’s Draft of the OpenID Federation Specification. This is a major step towards having the specification become final.

The previous Implementer’s Draft was in 2021. A lot has happened since then, largely motivated by feedback from actual implementations and deployments. Some highlights of progress made in the spec since then are:

Changed name from OpenID Connect Federation to OpenID Federation, since Federation can be used for trust establishment for any protocol (including OpenID Connect). Introduced distinct Federation endpoints. Clearly defined and consistently used the terms Entity Statement, Entity Configuration, and Subordinate Statement. Clearly defined which claims can occur in which kinds of Entity Statements. Clearly defined Entity Types and the Federation Entity entity type. Enhanced description of Trust Mark issuance and usage. Defined relationship between metadata and metadata policy. Clearly defined interactions between policy operators. Defined where constraints may occur. Tightened descriptions of Automatic Registration and Explicit Registration. Added Historical Keys. Defined and used trust_chain JWS Header Parameter. Allowed Trust Chains to start with non-Trust Anchors. Clarified use of client authentication. Used OAuth Protected Resource Metadata. Consistent error handling. Added General-Purpose JWT Claims section. Comprehensive use of content types and media types. IANA registration of parameters, claims, and media types. Added and improved many diagrams. Substantial rewrites for increased consistency and clarity. Added Giuseppe De Marco and Vladimir Dzhuvinov as editors.

As a preview of coming attractions, I’ll note that profiles of OpenID Federation are being written describing how it being used in wallet ecosystems and how it is being used in open finance ecosystems. And we’re creating a list of implementations. Watch this space for future announcements.

Special thanks to all the implementers and deployers who provided feedback to get us to this point!

Wednesday, June 12, 2024

This being our last day in Perth, we had a leisurely start, although that was limited by the need to check out of our hotel room on time. We had breakfast in the hotel, finished packing, and checked our bags for the day at the front desk.

Entrance to WA Museum Boola Bardip

We began with a visit to the WA Museum Boola Bardip (Boola Bardip means Many Stories in the local indigenous language), which is located in an area designated as Perth’s cultural center, with theaters and performance venues as well as the Western Australia library and archives. The extensive museum has exhibit halls dedicated to wildlife, cultural, and historical themes. The exhibits, many of them interactive, were well-designed and informative.

It was lunchtime after that, and Kenna remembered a “healthy burgers” chain, Grill’d, that we tried and liked during last spring’s trip to Brisbane. We split a lamb burger, which was a nice change of pace.

We stopped by the Perth Town Hall but couldn’t see much because of an event. The Town Hall is the second-oldest building in Perth, so the receptionist suggested we check out the oldest building, the Old Court House Law Museum. The main room of the building was a small courtroom set up as it would have been in Perth’s early days. The museum also had several exhibits about the evolution of laws applied to Aboriginal people and women, which showed how much things have changed since the beginning of the 20th century and how far we still need to go.

Perth Mint

We continued on to the Perth Mint, where we took a short self-guided tour that explained the Mint’s history and how it related to the Australian gold mining industry. It was fun talking with one of the “coin geeks” there. Today, I learned that successive British monarchs face opposite directions when depicted on coins. Queen Elizabeth II faces right, but King Charles faces to the left when he appears on Australian coins.

We then went to Elizabeth Quay for a break and a small snack and watched our final Perth sunset of the trip from there. We walked back to the hotel, collected our bags, and took a taxi to the airport. Unfortunately, we arrived about an hour before check-in opened, but we waited that out, passed through security, and caught our 11:45 flight to Melbourne. After a short layover we caught a nonstop to San Francisco, a 13-hour flight but a comfortable one. It was good to get home, early on June 13. Now for the challenge of staying awake, to get onto California time!

This article is the final installment in a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

Tuesday, June 11, 2024

We have heard a lot about Rottnest Island, which is just off the coast of Perth. Rottnest (“Rotto” to the locals) is a recreational island with cabins for rent, several beaches with snorkeling (in the summer), restaurants, a golf course, skydiving, and more. It is also the home of about 10,000 quokkas. Quokkas are small marsupials billed as “the happiest animals on earth.”

We booked a round trip on the Rottnest Express last night, so we got an early start this morning and grabbed coffee and a breakfast wrap on the way to the pier. We got seats upstairs for both the view and the airflow. The trip started slow (about 10 knots) as we left the pier, increased to about twice that once we reached a broad section of the Swan River, and slowed again as we approached Fremantle. We stopped, picked up more passengers there, and then proceeded to Rottnest Island at a higher speed.

Although we had been equipped with a map, we weren’t sure where to go when we got to Rottnest. We stumbled on the main village and were heading out when we noticed some rain starting to fall. We returned to the village to get some snacks, and the sky opened up: it poured! We were thankful to be in the village as there weren’t a lot of other places to get out of the rain. While there, we noticed our first quokka and gates on the grocery entrance and other businesses designed to keep quokkas out.

Despite a lot of public education not to feed the quokkas, many of them hang out in the village to eat up the crumbs and other droppage from human diners. They are indeed cute, but the “always smiling” thing seemed a little overstated. They can be a little aggressive, too: when Kenna and I shared a chicken sandwich for lunch, a quokka climbed on the bench between us and tried to get to the sandwich. We moved rather quickly.

Osprey Nest at Cape Vlamingh, Rottnest Island

The rain continued off and on, limiting our ability to venture beyond the village until we boarded a bus for a “discovery” tour we had booked for the afternoon. The tour took us around the perimeter of the island, with excellent descriptive commentary from a guide on the bus. We stopped at several points to take pictures, most notably at the far west end facing the vast Indian Ocean, where there was also a large osprey nest.

Following our tour, we stopped briefly to pick up souvenirs (a t-shirt for me), and it was time for our return trip to Perth. Along the way, in one of the slower-speed segments, a couple of hydrofoil surfers managed to “draft” off our ship, keeping up for quite a long time. We were also tailed briefly by a few dolphins.

Returning to our hotel, we were exhausted but managed to venture out for a quick pizza dinner.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.

👋 Hi, this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. To get issues like this in your inbox, sign up here:

Subscribe now

Unless you were under a rock since last week, you likely heard about the CrowdStrike / Windows outage that took down critical services like airlines, banks, supermarkets, police departments, hospitals, TV channels, and more, around the world. Businesses saw their Windows machines crash with the “Blue Screen of Death,” and no obvious fixes – at least not initially. The incident was unusual in size and scale, and also because it involved software running at the kernel-level; a factor which gives this us all the more reason to take a look at it.

Today, we cover:

Recap. 8.5M Windows machines impacted across industries

Root cause. An update to naming rules for finding malicious processes somehow resulted in the CSAgent.sys process attempting to write to an invalid memory address, thus crashing the operating system

A very slow, manual fix. Four days after the outage, recovery was ongoing, as every single impacted machine and host had to be fixed manually

Who’s responsible? Obviously, CrowdStrike is and it’s tempting to think Microsoft should share blame. A regulation from 2009 could also have played a role

Learnings for software engineers. Quantify potential impact, do canarying/staged rollouts, treat configuration like code, and more

Note: this is the last issue before The Pragmatic Engineer goes on summer break. There will be no The Pulse on Thursday, and no new issues next week. We return on Tuesday, 6 August. Thanks for your continuing support of this publication!

1. Recap

Last Friday (19 July,) the largest-ever software-initiated global outage hit machines worldwide. Millions of Windows 10 and 11 operating systems used by societally-critical businesses like airlines, banks, supermarkets, police departments, hospitals, TV channels, etc, suddenly crashed with the dreaded “Blue Screen of Death,” and no obvious way to fix them. This was a truly global outage; the US, Europe, Asia, South America, and Australia, were all hit. 

Global air travel descended into chaos, and in Alaska the emergency services number stopped working. In the UK, Sky News TV was unable to broadcast, and McDonalds had to close some of its Japanese outlets due to cash registers going down. In total, tens of thousands of businesses and millions of people were impacted. Meanwhile, in the world of Formula One racing, the Mercedes team saw its computers crash at the Hungarian grand prix. Ironically, one of the team’s sponsors is… CrowdStrike. Some photos of the outage in the wild:

Conveyor belt screens at LaGuardia airport, New York, on 19 July 2024. Source: Wikipedia Disneyland Paris was impacted, where staff switched to paper printouts to display wait times for rides. Source: The Points Guy Self-service checkout in a supermarket in New Zealand (Auckland.) Source: The New Zealand Herald The Windows crash caused by the CrowdStrike update caused issues for the F1 team sponsored by CrowdStrike. Source: BBC / Getty

All the business victims of this mega crash were customers of cybersecurity company CrowdStrike, which is the market leader in “endpoint security,” with around 20% market share. It installs software on Windows / Linux / Mac machines, and runs antivirus, firewalls, intrusion detection and prevention systems (IDP,) among others. What unleashed the global carnage was a single update by Crowdstrike to its ‘Falcon’ product.

We know 8.5M Windows machines were impacted globally from Microsoft sharing this number, later confirmed by CrowdStrike. Worst-hit of all might be Delta airlines, where around a third of flights (5,000) were canceled in three days. Even on day 4, Delta had to cancel another 1,000 flights as it recovered, and is on the hook for cash refunds for impacted customers.

2. Root cause

A few hours after Windows machines running CrowdStrike’s software started crashed, the company issued an update:

“CrowdStrike is actively assisting customers affected by a defect in a recent content update for Windows hosts. Mac and Linux hosts were not impacted. The issue has been identified and isolated, and a fix has been deployed. This was not a cyberattack.”

What happened is the company pushed out a “content” file (a binary file) to all customers at once, which then crashed the operating system. But how did it happen? As the incident was ongoing, some devs attempted to reconstruct what happened. Here are details from Patrick Wardle:

1. The process that crashed Windows is called “CSAgent.sys” 

2. The instruction that crashed is the Assembly instruction “mov  r9d, [r8].” This instructs to move the bytes in the r8 address to the r9d one. The problem is that r8 is an unmapped address (invalid), and so the process crashes!

The culprit: assembly instructions that crashed Windows machines across the globe. Source: Patrick Wardle on X

3. The crash was caused by the CSAgent.sys process reading a new “content” file CrowdStrike pushed to all clients called “C-00000291-*.sys” (where * can have additional characters.) Something went wrong related to this file and the parsing of it.

A day later, CrowdStrike shared more details:

1. The goal of the update was to detect maliciously-named pipes. CrowdStrike’s Falcon product observes how processes communicate on a machine, or across the network, to try and pinpoint malicious activity. The update was adding a new rule file to filter for suspiciously named pipes. A named pipe in the Windows world is a “named, one-way or duplex pipe for communication between the pipe server and one or more pipe clients.” These pipes can be used for inter-process communication (two processes talking to each other; here’s an example of processes sensing files between one another, or to communicate over the network. Named pipes are a common concept with operating systems for interprocess communication: Unix also uses this concept.

2. Released a new configuration file with new rules/naming. CrowdStrike calls config files that define behavior rules, like names for suspicious names pipes, “Channel files.” They store all these channel files in the location C:\Windows\System32\drivers\CrowdStrike\. These are numbered files, and the rules for named pipes are under number 291. Therefore, every file with the naming pattern “C-00000291-*.sys” is a rule for this category.

CrowdStrike released a new naming file in the update.

3. An unhandled error crashed the process and the operating system. While I’m itching to know about what the error actually was, CrowdStrike has only shared a very brief summary:

“The configuration update triggered a logic error that resulted in an operating system crash. This is not related to null bytes contained within Channel File 291 or any other Channel File.”

So, somehow, parsing these new naming rules resulted in an Assembly-level instruction that tries to move a memory location to an invalid location. This is what made Windows devices crash everywhere.

3. A slow, manual fix

Mitigating this outage was a lot more complicated than usual because a simple revert was insufficient. IT staff had to physically access each individual machine:

Fixing an impacted Windows machine because there was no remote fix option. Source: techAU on X

CrowdStrike posted mitigation steps for IT admins and developers wanting to get themselves unblocked, a few hours after the incident. The steps were:

Boot Windows into Safe Mode or the Windows Recovery Environment

Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

Locate the file matching “C-00000291*.sys” and delete it

Boot the host

The recovery process might need a local administrator on a machine with the right to delete the offending file. The steps are specialized enough that regular users would struggle to perform the recovery: and so at most companies it’s up to IT staff to manually fix every machine. Plus, at many places all Windows laptops were impacted. An IT admin shared a glimpse of the task, posting an image of 120 of 2,000 laptops to be fixed in one weekend, ideally!

Some laptops to be reset. Source: Dunken K Bliths on X

As software engineers, when we see a highly manual process our first thought is whether we can automate it, or do it faster in a clever way. With 8.5M machines needing resets, it’s obvious a manual process is incredibly time consuming. So independent developers, and also Microsoft, stepped in:

iOS developer and Windows tinkerer Adam Demasi created the Unistrike tool a day later. With some additional setup, you can create a USB stick to plug into each impacted machine for faster recovery

Microsoft also released a similar recovery tool the day after the outage

Two days later, CrowdStrike shared that they were testing a new, faster recovery technique for customers

Four days after the outage, there still remained plenty of the 8.5M impacted Windows devices that still weren’t fixed. It turns out that crashing operating systems at scale is a lot harder to recover at scale than applications, for which patches can be sent out to clients (mobile and desktop apps,) or when the fix can be done server side (services, backend applications, web apps.)

4. Who’s responsible?

It was a little amusing that the news initially reported this as a “Microsoft outage” or a “Windows outage” because it’s a bit distant from the facts. So who “owns” the world’s biggest-ever software crash?

CrowdStrike – obviously

Needless to say, most blame lies with CrowdStrike. We can only speculate at this point about which areas were skipped, or done insufficiently thoroughly. Hopefully, we will learn more in a public-facing postmortem. Meantime, here are some questions that CrowdStrike should be asking – and, most likely, is:

1. Was the change tested, and how? Was the change in this config file (C-00000291-*.sys) tested in manual and automated scenarios? If so, how did the tests pass, and why did the crash happen in production? A more interesting question that only CrowdStrike can answer is how the configs were tested in an automated way; and indeed, were they? We know testing environments can never replicate production in full, so it’s expected that bugs can make their way through tests, undetected.

2. Were these config changes dogfooded? Was this change rolled out to CrowdStrike staff, before release to the public? If yes, did some CrowdStrike employees also see their OSs crash? If yes, then why did the rollout proceed? If there was dogfooding, but no employees’ machines crashed; an interesting question is: why not?

3. Was there a canary rollout? We cover the topic of canarying in Shipping to Production:

“Canarying” comes from the phrase “canary in the coal mine.” In the early 20th century, miners took caged canary birds with them underground. The bird has a lower tolerance for toxic gasses than humans do, so if the bird stopped chirping or fainted, it was a warning sign to miners that gas was present, and for them to evacuate.

Today, canary testing means rolling out code changes to a smaller percentage of the user base, then monitoring the health signals of this deployment for signs that something’s not right. A common way to implement canarying is to either route traffic to the new version of the code using a load balancer, or to deploy a new version of the code to a single node.”

Canarying is a subset of staged rollouts:

“Staged rollouts mean shipping changes step by step, evaluating the results at each stage before proceeding. Staged rollouts typically define the percentage of the user base to get the changed functionality, or the region where this functionality should roll out, or both.

A staged rollout plan may look like this:

Phase 1: 10% rollout in New Zealand (a small market to validate changes)

Phase 2: 50% rollout in New Zealand

Phase 3: 100% rollout in New Zealand

Phase 4: 10% rollout, globally

Phase 5: 25% rollout, globally

Phase 6: 50% rollout, globally

Phase 7: 100% rollout, globally

Between each rollout stage, a criteria is set for when the rollout can continue. This is typically defined as there being no unexpected regressions and the expected changes to business metrics being observed.”

Did CrowdStrike use these approaches, or was it more of a “YOLO rollout,” where the configuration file was pushed to all customers at the same time? Right now, we don’t know.

From the incident response communication, it sounds like the change was more a “YOLO rollout” because the changed file was labeled as “content,” not business logic. This is despite it containing rules on how to detect named pipes, which you could argue is business logic that should be rolled out in phases, not all at once!

4. Does CrowdStrike assume that binary (“content”) files cannot break software running at kernel level? Common rollout strategies for shipping code were likely absent when shipping these new configuration files. Did CrowdStrike assume – implicitly or explicitly – that these “content” files could not crash the process?

CrowdStrike’s software operates at the kernel level in Windows, meaning its process is operating with the highest level of privileges and access in the OS. This means it can crash the whole system; for example, by corrupting part of the OS’s memory. CrowdStrike operating at this level is necessary for it to oversee processes running across the OS, and to discover threats and vulnerabilities. But this also means that an update – even an innocent-looking content file! – can cause a crash. 

5. Did the company ignore a previous similar outage? A Hacker News commenter working at a civic tech lab shared that, a few months ago, CrowdStrike caused a similar outage for their Linux systems. This dev summarized:

“Crowdstrike did this to our production linux fleet back on April 19th [2024], and I've been dying to rant about it.

The short version is we're a civic tech lab, so have a bunch of different production websites made at different times on different infrastructure. We run Crowdstrike provided by our enterprise. Crowdstrike pushed an update on a Friday evening that was incompatible with the up-to-date Debian stable. So we patched Debian as usual, everything was fine for a week, and then all of our servers across multiple websites and cloud hosts simultaneously hard crashed and refused to boot.

When we connected one of the disks to a new machine and checked the logs, Crowdstrike looked like a culprit, so we manually deleted it, the machine booted, tried reinstalling it and the machine immediately crashed again. OK, let's file a support ticket and get an engineer on the line.

Crowdstrike took a day to respond, and then asked for a bunch more proof (beyond the above) that it was their fault. They acknowledged the bug a day later, and weeks later had a root cause analysis that they didn't cover our scenario (Debian stable running version n-1, I think, which is a supported configuration) in their test matrix. In our own post mortem there was no real ability to prevent the same thing from happening again: "we push software to your machines any time we want, whether or not it's urgent, without testing it," seems to be core to the model.”

These details suggest that CrowdStrike could or should have been aware that it can – and does – crash kernel processes with updates. If so, the obvious question is why this outage did not serve as a warning to tweak the rollout process, as opposed to just improving testing? 

In fairness, a company like CrowdStrike has hundreds of engineering teams, and one team observing an outage is information that will not necessarily spread through the organization. Still, the CrowdStrike process crashing the OS was surely a known vulnerability, as it’s the most obvious way to brick a customer’s machine which it is meant to defend.

Microsoft / Windows?

Why can CrowdStrike run processes at kernel level which can crash an operating system? After all, Apple made changes to MacOS to run third-party software at user level, not kernel. From Electric Light, in 2021:

“For some years now, Apple has been encouraging third-party developers to move away from kernel extensions to equivalents which run at a user level rather than in Ring 1. However, it has only been in the last year or so that Apple has provided sufficient support for this to be feasible. Coupled with the fact that M1 Macs have to be run at a reduced level of security to be able to load third-party kernel extensions, almost all software and hardware which used to rely on kernel extensions should now be switching to Apple’s new alternatives such as system extensions. This article explains the differences these make to the user.”

So on Mac, the same CrowdStrike process would run in the user space, and if it crashes it would not take the whole system down with it.

However, on Windows and Linux, antivirus and other cybersecurity software usually runs at the kernel level, and always has done. So why hasn’t Microsoft followed Apple’s approach and banned third parties from the kernel space? Turns out that a Symantec complaint in the 2000s, and EU regulation, played a role. 

Regulation to blame?

The Wall Street Journal asked Microsoft why it won’t limit third-party software like CrowdStrike to run only in the user space, not the kernel space. Its response:

“A Microsoft spokesman said it cannot legally wall off its operating system in the same way Apple does because of an understanding it reached with the European Commission following a complaint. In 2009, Microsoft agreed it would give makers of security software the same level of access to Windows that Microsoft gets.”

Ironically, all of this started in 2006 with Microsoft wanting to make its kernel more secure for Windows Vista. From CIO.com at the time (emphasis mine):

“Security vendors like Symantec are in a state of heightened sensitivity these days as they’ve begun to compete with Microsoft head-on, and the specter of further antitrust actions looms over Microsoft’s every move in the security space. Last week, the European Union’s spokesman on competition, Jonathan Todd, warned that the market could be threatened if Microsoft doesn’t allow security vendors a fair chance of competing.

Symantec and other security vendors dislike PatchGuard because it prevents them from accessing the Windows kernel. They say it will stop them from delivering important features like Symantec’s “anti-tampering” technology, which prevents malicious programs from modifying Symantec’s own software.

PatchGuard will also make it more difficult for security vendors to protect against malicious software that takes advantage of kernel-level bugs, said Eric Sites, vice president of research and development with Sunbelt Software. (...)

Microsoft declined to be interviewed for this article, but in an interview with IDG News last week a Microsoft executive said that PatchGuard was simply an effort to prevent the kernel from being misused.

“We think that there’s a significant amount of confusion around… certain security features in the product that we think raise the foundation,” said Stephen Toulouse, a senior product manager in the Security Technology Unit. “What we’re doing is we’re walling off the kernel from attackers, because the functionality that is currently there was never meant to be used by anybody — by software vendors or attackers.”

In the end, Symantec and other vendors won. Microsoft could only “ban” security vendors from running in the kernel space if it also did not run its own security software there. So while Microsoft could be seen as partly responsible for this crash, the company had little choice in the actions which created the circumstances for it to happen! 

There would have likely been a way, though: if Microsoft moved their own security solution – such as Windows Defender – out of the kernel space, closing it off to all security vendors, including itself. Doing so would likely mean a large enough re-architecture of the Windows security stack. It would also limit the capabilities of third-party vendor solutions, and any such change would trigger outcry and more complaints to regulators by security vendors. It would be no different to the complaints and escalations of 2006, when Vista attempted to lock vendors out of the kernel space.

5. Learnings for software engineers

Here are some learnings that us software engineers can take from this incident, as things stand:

Quantify the impact of software crashing everywhere

What happens if your company’s product crashes irrecoverably for a couple of hours? Ignore the fact that this seems so unlikely as to be impossible – because it has just happened to CrowdStrike. If it happened, what would the impact be on your company and the outside world? For example:

If Amazon crashed worldwide for a few hours, sellers would lose revenue and a subset of shoppers could lose access to essential items. Amazon would lose revenue and suffer reputational damage.

If TikTok crashed for hours worldwide for a few hours, brands would not be able to serve ads, and customers would feel indifferent, slightly annoyed, or furious about not being able to use the social platform. Far-fetched theories could emerge about TikTok being blocked, the company would lose ad revenue, and users would temporarily flock to alternatives like Instagram Reels and Snap.

If a major phone and internet carrier crashed, the impact would be far worse than the two above, combined. Businesses would struggle to operate and emergency services could be impacted. The damage would be reputational, lasting, and government intervention could also follow. Last November, we covered what happened when Half of Australia’s internet was cut off for 14 hours

This exercise is helpful because it can give a sense of how expensive an outage could be. Knowing the “blast radius” can help get buy-in to make systems more resilient, and make it quicker to detect and mitigate incidents.

Review how things reach production

What needs to happen for a code or asset change to be shipped to all customers? We go deep on this topic in Shipping to Production. As a recap, here are two extremes in shipping to production: CrowdStrike seems to have chosen the “YOLO” option for this change, and it cost them dearly:

Two “extremes” of shipping to production: YOLO and thorough verification. Source: Shipping to Production Do canarying / staged rollouts

If your software crashing everywhere has a big enough “blast radius” to make a failure unacceptable, then do not roll out changes to all customers at once! Do a canary or a staged rollout.

It’s true that canarying and staged rollouts are overkill for certain products, like those with few users, or which do not generate revenue, or are experimental. Setting up canarying or staged rollouts is effort and does slow down rollout. But if your product is used by many people, or is critical enough, then this rollout strategy is non-negotiable. Take it from former Google Chrome engineer Marc-Antoine Ruel:

“From the get go, Google Chrome had 3 release channels plus nightly builds:

canary (nightly), dev, beta, stable. Each increases the blast radius.

Dev did get broken builds. Beta broke in subtle ways. This release scheme reduced the blast radius.

Net result: 126 stable releases!”

Treat configuration like code

Staff engineer Jacques Bernier formerly worked at Twitch and has shared how Amazon treated code changes:

“The Amazon build and deploy system is one that I miss deeply. It was so powerful. And it considered all changes equal. It included code, dependencies all the way down to the operating system and infrastructure in the same build and deploy pipeline with multiple stages.

Config change is code. Infrastructure change is code change. Dependency update is code change. Code change is code change. It is all the same.

Staged rollout is one of the best ways (if not the best) to reduce blast radius.”

What changes do dependencies/vendors “silently” push?

CrowdStrike broke most customers’ businesses because it silently and automatically shipped business logic changes. Even if customers wanted to “block” a change, or only allow it for a subset of machines at first, they could not.

It’s a good reminder that software can be broken not just by code, but by your dependencies or vendors. So now is a good time to consider these questions:

How are dependencies (libraries, frameworks) updated? Are they done automatically, or manually? This is especially relevant when working with package managers that might get these automatically.

What about vendor dependencies – SDKs or APIs? Are you the one making changes, or are vendors pushing silent changes?

List all the points which could be affected by a faulty “silent” change from a third-party you use and (currently) trust.

An outage is no one person’s fault

It’s easy to blame whoever wrote an offending piece of code for a crash; perhaps an intern lacking experience, or a veteran engineer having a bad day. But pointing the finger of blame at individuals is the wrong approach. Microsoft Veteran Scott Hanselman summarizes why a failure at this scale is never one person’s fault (emphasis mine:)

“Here’s the thing folks. I’ve been coding for 32 years. When something like this happens it’s an organizational failure. Yes, some human wrote a bad line. Someone can “git blame” and point to a human, and it’s awful. 

But it’s the testing, the Cl/CD, the A/B testing, the metered rollouts, an “oh s**t” button to roll it back, the code coverage, the static analysis tools, the code reviews, the organizational health, and on and on. 

It’s always one line of code but it’s NEVER one person. Implying inclusion policies caused a bug is simplistic, reductive, and racist. Engineering is a team sport. Inclusion makes for good teams. Good engineering practices make for good software. Engineering practices failed to find a bug multiple times, regardless of the seniority of the human who checked that code in. 

Solving the larger system thinking SDLC matters more than the null pointer check. This isn’t a “git gud C++ is hard” issue and it damn well isn’t an DEI one.”

Takeaways

Widespread outages are always bad, but one upside is that they force us engineers to pause and reflect:

Can something similarly catastrophic happen at my company, and if so, how?

What would the impact be in my situation?

What do we do to avoid being the “next CrowdStrike?”

There’s no better time than now to make a case to your leadership for investing properly in reliability. The CrowdStrike outage is now officially the largest-ever software outage on the planet, and customers have suffered heavy financial and reputational damage. The financial loss is still unclear for CrowdStrike, but you can assume it will be huge, as some businesses will seek compensation for the damage done.

For CrowdStrike, the reputational damage could hardly be worse. Until a few days ago, the company was the gold standard in endpoint security compliance. No longer: its name is linked with the biggest outage anybody’s seen. After such a high-profile blunder that reveals the company had no staged rollout processes in place for business rule changes (“channel files,”) the reputation of Crowdstrike has suffered a hit which it will take a long time to recover from.

No business wants such a blow from a single bad deploy, but it’s happened. If you see gaps in your company’s release processes – testing, rollout, monitoring, alerting, etc – then now is the time to take your concerns and suggestions to the table! Talk with your manager or skip-level; they will be more likely to champion ideas which make production systems resilient.

CrowdStrike is certain to learn its lesson, and doubtless its future release processes will be world class. Good luck to the team there (and teams at all affected customers) for mitigating the outage, and for work ahead at CrowdStrike to overhaul internal processes.

Let’s hope many companies follow suit, so this historic event ends up being a net positive learning experience for the tech industry.

We’ve previously covered outages with interesting learnings. Check out these for more analysis and learnings, and for making systems more reliable:

Three cloud providers, three outages: three different responses 

AWS’s us-east-1 outage

Inside Datadog’s $5M outage

Half of Australia knocked offline for 14 hours

Inside the longest Atlassian outage 

Four failed attempts to determine why Roblox was down for three days

If you’re not a full subscriber to the newsletter, you missed the issue on AI tooling reality check for software engineers, AWS shifting its focus away from infra, the Trimodal nature of tech compensation revisited, and more. Subscribe to get two full issues every week. Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

Subscribe now

Update on 23 July: I interpreted the mov instruction parameters in an incorrect order. Thanks to Hugh Brown in the comments for pointing this out!

President Joe Biden, grappling with dementia, has finally stepped aside, leaving Vice President Kamala Harris to run in his place for re-election. This is the breath of fresh air Dems…

Monday, June 10, 2024

Kenna and I slept in a bit today, then set out into downtown Perth. We grabbed breakfast at one of the many coffee shops. Perth seems to be quite a coffee city, but there are no Starbucks here.

We walked to Elizabeth Quay and the Barrack Street jetty, from which many tour boats originate. We picked up some information on Rottnest Island tours and were surprised when Lee and Colleen, two of our fellow passengers on the Orion, also showed up. We may see them again.

From there, we walked to Kings Park and its botanical garden. We made some wrong turns but eventually found it and discovered it is up quite a hill from downtown and has a great view. The botanical garden is nicely labeled and includes specific sections for areas like the Kimberley. From a distance, we saw Lee and Colleen here too. We could have spent a lot more time here, but we wanted to make it out to Fremantle, so we caught a bus to the train station and a train to Fremantle.

One of our first goals in Fremantle (“Freo” to the locals) was to visit the Indian Ocean for the first time. We went out to a nearby beach and touched the water, which, of course, felt like…water. Soon after that, we ran into a couple of other fellow passengers from the Orion, who had just returned from the Maritime Museum. They told us it was excellent, so we headed over there.

After a quick late lunch in their snack bar, we toured the exhibits. One memorable exhibit for me was Australia II, the Australian winner of the America’s Cup sailing race in 1983. It was a big deal at the time and continues to be a source of pride for Australians.

Fremantle hadn’t initially seemed much like other beach towns (Santa Cruz, Brighton UK, etc.) I have visited. But as we headed more into the city center, we began to see the makings of a beach town: some brewpubs, a game arcade, souvenir shops, etc. It was still somewhat more restrained than many.

Exhausted from several miles of walking, we returned to the train station and rode back to Perth, where we had more walking to do to get to the hotel. We opted to have dinner in the hotel’s pub, which was mediocre and surprisingly closed at 8 p.m. — earlier than most any pub I have seen.

This article is part of a series about our recent trip to Indonesia and Australia. To see the introductory article in the series, click here.