Last Update 10:44 AM July 30, 2021 (UTC)

Identity Blog Catcher

Brought to you by Identity Woman and Infominer.
Support this collaboration on Patreon!!!

Friday, 30. July 2021

John Philpin : Lifestream

Apple’s Home App … do not get me started. They should fir

Apple’s Home App … do not get me started. They should fire the team.

Apple’s Home App … do not get me started.

They should fire the team.


”Create an environment that tolerates misfits, rewards act

”Create an environment that tolerates misfits, rewards actual creative behaviors, and embraces people following their own interests. Companies that don’t will suffer the fate of Ford, IBM, and the other former giants that focused on short-term efficiency at the cost of long-term innovation.” Stowe Boyd (I think)

”Create an environment that tolerates misfits, rewards actual creative behaviors, and embraces people following their own interests. Companies that don’t will suffer the fate of Ford, IBM, and the other former giants that focused on short-term efficiency at the cost of long-term innovation.”

Stowe Boyd

(I think)


Creator Platforms hmmm ….

Creator Platforms hmmm ….

Creator Platforms

hmmm ….


If you are having a meeting with someone with an hourly rate

If you are having a meeting with someone with an hourly rate - then you know how much they charge by the minute (’R’). New Rule : [M(n) =number of minutes you keep me waiting.] * R = Discount on Bill

If you are having a meeting with someone with an hourly rate - then you know how much they charge by the minute (’R’).

New Rule :

[M(n) =number of minutes you keep me waiting.] * R = Discount on Bill


“This time around it was Windows’ moment to shine and in t

“This time around it was Windows’ moment to shine and in the next update I’ll share all the details on 1Password 8 for Mac and how to join the early access.” Dave Teare - 1Password Founder Must be that new ‘Corporate First’ mentality that his new investors brought in.

“This time around it was Windows’ moment to shine and in the next update I’ll share all the details on 1Password 8 for Mac and how to join the early access.”

Dave Teare - 1Password Founder

Must be that new ‘Corporate First’ mentality that his new investors brought in.

Thursday, 29. July 2021

John Philpin : Lifestream

Scarlett Johansson Sues Disney Over ‘Black Widow’ Streaming

Scarlett Johansson Sues Disney Over ‘Black Widow’ Streaming Release Finally the film world begins to wrap its head around the issues that have plagued musicians since the MP3.

Scarlett Johansson Sues Disney Over ‘Black Widow’ Streaming Release

Finally the film world begins to wrap its head around the issues that have plagued musicians since the MP3.


Simon Willison

Weeknotes: datasette-remote-metadata, sqlite-transform --multi

I mentioned Project Pelican (still a codename until the end of the embargo) last week. This week it inspired a new plugin, datasette-remote-metadata. I also published an article describing the Baked Data architectural patttern and shared the video and transcript of a talk I gave at PyGotham last year. datasette-remote-metadata Datasette's metadata system is one of the key ways of customizing

I mentioned Project Pelican (still a codename until the end of the embargo) last week. This week it inspired a new plugin, datasette-remote-metadata.

I also published an article describing the Baked Data architectural patttern and shared the video and transcript of a talk I gave at PyGotham last year.

datasette-remote-metadata

Datasette's metadata system is one of the key ways of customizing a Datasette instance: it lets you add extra information about the tables hosted by the instance, lets you configure canned queries and also (due to some scope creep that I'd love to clean up) lets you configure options for various Datasette plugins.

There's just one catch: when you update the metadata for an instance, you need to re-deploy the whole thing. I increasingly find myself using Datasette and Cloud Run to host databases that are over 1GB in size, so re-deploying just to tweak some text on the homepage is less than ideal.

The datasette-remote-metadata plugin uses the new get_metadata() plugin hook from Datasette 0.58 to work around this problem.

Essentially, it lets you tell Datasette "load your metadata from this external URL". You provide it with a URL to a JSON or YAML file hosted elsewhere and the plugin will load that file on startup and periodically (by default every 30 seconds) check to see if it has changed.

Those checks for updates only happen if Datasette is receiving traffic. Datasette will try for up to 200ms, then continue the fetch but serve the request using the now-stale metadata rather than block for longer than that.

I'm trying this out for Project Pelican and it seems to be working great - which means we can iterate on the metadata (including the description_html block displayed on the Datasette homepage) without needing to constantly re-deploy the 700+ MB SQLite database file to Cloud Run.

sqlite-transform --multi

sqlite-transform is my command-line tool for running transformations against columns of data in a SQLite database - parsing dates, splitting comma-separated lists into JSON arrays or using custom Python code to apply any custom transformation.

The latest release adds support for splitting one input column into multiple output columns. This is best explained with an example: consider a database table with a location column that contains latitude,longitude points which you would like to split out into separate latitude and longitude columns.

sqlite-transform lambda demo.db places location \ --code 'return { "latitude": float(value.split(",")[0]), "longitude": float(value.split(",")[1]), }' --multi --drop

The new --multi option means that the passed in --code will return a Python dictionary, and the tool should then add new columns to the specified table for each of the returned dictionary keys and populate them with the corresponding values.

The new --drop option tells the tool to drop that location column after the transformation has been completed.

Releases this week datasette-remote-metadata: 0.1 - (2 releases total) - 2021-07-29
Periodically refresh Datasette metadata from a remote URL datasette-haversine: 0.2 - 2021-07-28
Datasette plugin that adds a custom SQL function for haversine distances datasette-publish-vercel: 0.10 - (16 releases total) - 2021-07-25
Datasette plugin for publishing data using Vercel sqlite-transform: 1.2 - (9 releases total) - 2021-07-25
Tool for running transformations on columns in a SQLite database sqlite-utils: 3.13 - (81 releases total) - 2021-07-24
Python CLI utility and library for manipulating SQLite databases TIL this week Extracting objects recursively with jq

Ben Werdmüller

She exposed how Facebook enabled global political manipulation. Now she's telling her story.

“Her story reveals that it is really pure luck that we now know so much about how Facebook enables election interference globally. Zhang was not just the only person fighting an entire swath of political manipulation, it also wasn’t her job. She had discovered the problem because of a unique confluence of skills and passion, then taken it upon herself, driven by an extraordina

“Her story reveals that it is really pure luck that we now know so much about how Facebook enables election interference globally. Zhang was not just the only person fighting an entire swath of political manipulation, it also wasn’t her job. She had discovered the problem because of a unique confluence of skills and passion, then taken it upon herself, driven by an extraordinary sense of moral responsibility. To regulators around the world considering how to rein in the company, this should be a wakeup call.”

[Link]


Simon Willison

Clickhouse on Cloud Run

Clickhouse on Cloud Run Alex Reid figured out how to run Clickhouse against read-only baked data on Cloud Run last year, and wrote up some comprehensive notes. Via @alexjreid

Clickhouse on Cloud Run

Alex Reid figured out how to run Clickhouse against read-only baked data on Cloud Run last year, and wrote up some comprehensive notes.

Via @alexjreid


John Philpin : Lifestream

15 posts from @leo in less than a minute - after a three yea

15 posts from @leo in less than a minute - after a three year gap .. somebody changed a server setting.

15 posts from @leo in less than a minute - after a three year gap .. somebody changed a server setting.


SuperMarkets 3/3 I wouldn’t judge American coffee by Star

SuperMarkets 3/3 I wouldn’t judge American coffee by Starbucks - so we shouldn’t judge American Supermarkets by Wholefoods and Safeways.

SuperMarkets 3/3

I wouldn’t judge American coffee by Starbucks - so we shouldn’t judge American Supermarkets by Wholefoods and Safeways.


SuperMarkets 2/3 If you think US supermarkets are expensi

SuperMarkets 2/3 If you think US supermarkets are expensive you should visit New Zealand.

SuperMarkets 2/3

If you think US supermarkets are expensive you should visit New Zealand.


SuperMarkets 1/3 The international section is really the

SuperMarkets 1/3 The international section is really the ‘other’ section.

SuperMarkets 1/3

The international section is really the ‘other’ section.

Wednesday, 28. July 2021

Margo Johnson

An Ode to Generous Networks

Reflecting on change and community in a time of personal transition. Staring out over a world of possibilities (Joshua Tree, July 2021) Navigating through this global pandemic has leveled up both my tolerance for ambiguity and my appreciation for community — including friends, family, co-workers, and global colleagues. These dual forces of change and connection are also guiding me forward as
Reflecting on change and community in a time of personal transition. Staring out over a world of possibilities (Joshua Tree, July 2021)

Navigating through this global pandemic has leveled up both my tolerance for ambiguity and my appreciation for community — including friends, family, co-workers, and global colleagues. These dual forces of change and connection are also guiding me forward as I embark on a new professional chapter.

After three deeply rewarding and challenging years with Transmute, I’m transitioning into a new role this fall.

I will continue to work in the decentralized identity and verifiable credentials ecosystem but will return to more of my impact roots as Senior Director of Product with Kiva’s Protocol team — part of an international non-profit working on digital identity as an enabler of deeper financial inclusion. I will also remain a champion and alumni advisor of Transmute, cheering the team on through next iterations of company and product development.

Making any big transition like this always surfaces the forest through the trees for me, bringing some longer term trends into focus in a way I often miss when heads down on day to day work.

In particular I’ve found myself reflecting on the philosophy and practice of open versus closed networks, and my growing conviction that open networks are a key to satisfaction and success, both for individuals and for businesses.

My working definition of “open networks” boils down to a mindset and style of engagement that assumes generosity, collaboration, and expanded connections will lead to mutual benefit. This can look like contributing to global standards and open source work, being proactive with connecting people, offering support without expectation of returned favors, and celebrating colleagues as they transition organizations (something I’ve been fortunate to experience this month with Transmute).

Open network interactions are empathetic and relational rather than transactional, rooted in a belief that we can progress and grow together, creating more opportunity in the process rather than competing for something finite.

By contrast, a more closed network approach (all too often the default) sees the gain of others as a personal loss if not compensated in some way. It is a more protective mentality that discourages transparency and informal exchange. It looks like vendor lock-in, punitive non-compete agreements, and proprietary code. To be clear, some degree of this is necessary for businesses to function today — to show value for investors, protect IP, drive competitive advantage, etc. However, taken as a default state, closed networks can crush both innovation and the spirits of individuals whose interests expand beyond the bounds of any single identity or organization.

One of the things that drew me to Transmute — and more broadly to emergent technology like decentralized identities and verifiable credentials — is that the open network style is often the default culture of engagement. In fact, this approach is necessary in the face of pre-competitive table stakes like technical interoperability as well as collaborative engagement and advocacy in the broader regulatory environment. Essentially, the adoption of this technology is far bigger than any one of us, and we have to work together to build the market and adoption we want to see in the world.

This open, generous network ethos aligns with the future I want to be part of, both with the organizations I help build and personally as a self-actualizing and social human. Part of the “crafted self” we talk about as identity workers includes the ability to be selectively multi-dimensional beings across different settings. To me this includes the freedom to traverse organizations and even industries as our interests take us, and to carry our reputation and relationships along that journey. Viewed in this way, the interoperability and autonomy values woven into the foundations of decentralized tech become further tools that can help bring to bear an even more generous world.

A huge thank you to the Transmute Team — particularly Karyl Fowler and Orie Steele — for leading the way, both as technologists and founders, in making the bet that movement across systems and organizations is a win for all parties involved (be they verifiable trade documents or prior employees). It has been a pleasure to work for and with you over the past few years.

***

If you are interested in some of the cross-network work happening in the decentralized tech space you can also check out the Decentralized Identity Foundation working groups, the Internet Identity Workshop, the Trust Over IP Foundation, and the Department of Homeland Security Silicon Valley Innovation Program (to name just a few!).

Jumping off, jumping in! (Mammoth Lakes CA, July 2021)

Simon Willison

The Baked Data architectural pattern

I've been exploring an architectural pattern for publishing websites over the past few years that I call the "Baked Data" pattern. It provides many of the advantages of static site generators while avoiding most of their limitations. I think it deserves to be used more widely. I define the Baked Data architectural pattern as the following: Baked Data: bundling a read-only copy of your data a

I've been exploring an architectural pattern for publishing websites over the past few years that I call the "Baked Data" pattern. It provides many of the advantages of static site generators while avoiding most of their limitations. I think it deserves to be used more widely.

I define the Baked Data architectural pattern as the following:

Baked Data: bundling a read-only copy of your data alongside the code for your application, as part of the same deployment

Most dynamic websites keep their code and data separate: the code runs on an application server, the data lives independently in some kind of external data store - something like PostgreSQL, MySQL or MongoDB.

With Baked Data, the data is deployed as part of the application bundle. Any time the content changes, a fresh copy of the site is deployed that includes those updates.

I mostly use SQLite database files for this, but plenty of other formats can work here too.

This works particularly well with so-called "serverless" deployment platforms - platforms that support stateless deployments and only charge for resources spent servicing incoming requests ("scale to zero").

Since every change to the data results in a fresh deployment this pattern doesn't work for sites that change often - but in my experience many content-oriented sites update their content at most a few times a day. Consider blogs, documentation sites, project websites - anything where content is edited by a small group of authors.

Benefits of Baked Data

Why would you want to apply this pattern? A few reasons:

Inexpensive to host. Anywhere that can run application code can host a Baked Data application - there's no need to pay extra for a managed database system. Scale to zero serverless hosts such as Cloud Run, Vercel or AWS Lambda will charge only cents per month for low-traffic deployments. Easy to scale. Need to handle more traffic? Run more copies of your application and its bundled data. Horizontally scaling Baked Data applications is trivial. They're also a great fit to run behind a caching proxy CDN such as Cloudflare or Fastly - when you deploy a new version you can purge that entire cache. Difficult to break. Hosting server-side applications on a VPS is always disquieting because there's so much that might go wrong - the server could be compromised, or a rogue log file could cause it to run out of disk space. With Baked Data the worst that can happen is that you need to re-deploy the application - there's no risk at all of data loss, and providers that can auto-restart code can recover from errors automatically. Server-side functionality is supported. Static site generators provide many of the above benefits, but with the limitation that any dynamic functionality needs to happen in client-side JavaScript. With a Baked Data application you can execute server-side code too. Templated pages. Another improvement over static site generators: if you have 10,000 pages, a static site generator will need to generate 10,000 HTML files. With Baked Data those 10,000 pages can exist as rows in a single SQLite database file, and the pages can be generated at run-time using a server-side template. Easy to support multiple formats. Since your content is in a dynamic data store, outputting that same content in alternative formats is easy. I use Datasette plugins for this: datasette-atom can produce an Atom feed from a SQL query, and datasette-ics does the same thing for iCalendar feeds. Integrates well with version control. I like to keep my site content under version control. The Baked Data pattern works well with build scripts that read content from a git repository and use it to build assets that are bundled with the deployment. How to bake your data

My initial implementations of Baked Data have all used SQLite. It's an ideal format for this kind of application: a single binary file which can store anything that can be represented as relational tables, JSON documents or binary objects - essentially anything at all.

Any format that can be read from disk by your dynamic server-side code will work too: YAML or CSV files, Berkeley DB files, or anything else that can be represented by a bucket of read-only bytes in a file on disk.

[I have a hunch that you could even use something like PostgreSQL, MySQL or Elasticsearch by packaging up their on-disk representations and shipping them as part of a Docker container, but I've not tried that myself yet.]

Once your data is available in a file, your application code can read from that file and use it to generate and return web pages.

You can write code that does this in any server-side language. I use Python, usually with my Datasette application server which can read from a SQLite database file and use Jinja templates to generate pages.

The final piece of the puzzle is a build and deploy script. I use GitHub Actions for this, but any CI tool will work well here. The script builds the site content into a deployable asset, then deploys that asset along with the application code to a hosting platform.

Baked Data in action: datasette.io

The most sophisticated Baked Data site I've published myself is the official website for my Datasette project, datasette.io - source code in this repo.

The site is deployed using Cloud Run. It's actually a heavily customized Datasette instance, using a custom template for the homepage, custom pages for other parts of the site and the datasette-template-sql plugin to execute SQL queries and display their results from those templates.

The site currently runs off four database files:

content.db has most of the site content. It is built inside GitHub Actions by the build.sh script, which does the following: Import the contents of the news.yaml file into a news table using yaml-to-sqlite. Import the markdown files from the for/ folder (use-cases for Datasette) into the uses table using markdown-to-sqlite. Populate the plugin_repos and tool_repos single-column tables using data from more YAML files. These are used in the next step. Runs the build_directory.py Python script. This uses the GitHub GraphQL API to fetch information about all of those plugin and tool repositories, including their README files and their most recent tagged releases. Populates a stats table with the latest download statistics for all of the Datasette ecosystem PyPI packages. That data is imported from a stats.json file in my simonw/package-stats repository, which is itself populated by this git scraping script that runs in GitHub Actions. I also use this for my Datasette Downloads Observable notebook. blog.db contains content from my blog that carries any of the datasette, dogsheep or sqliteutils tags. This is fetched by the fetch_blog_content.py script, which hits the paginated per-tag Atom feed for my blog content, implemented in Django here. docs-index.db is a database table containing the documentation for the most recent stable Datasette release, broken up by sections. This database file is downloaded from a separate site, stable-docs.datasette.io, which is built and deployed as part of Datasette's release process. dogsheep-index.db is the search index that powers site search (e.g. this search for dogsheep). The search index is built by dogsheep-beta using data pulled from tables in the other database files, as configured by this YAML file.

The site is automatically deployed once a day by a scheduled action, and I can also manually trigger that action if I want to ensure a new software release is reflected on the homepage.

Other real-world examples of Baked Data

I'm currently running two other sites using this pattern:

Niche Museums is my blog about tiny museums that I've visited. Again, it's Datasette with custom templates. Most of the content comes from this museums.yaml file, but I also run a script to figure out when each item was created or updated from the git history. My TILs site runs on Vercel and is built from my simonw/til GitHub repository by this build script (populating this tils table). It uses the GitHub API to convert GitHub Flavored Markdown to HTML. I'm also running a script that generates small screenshots of each page and stashes them in a BLOB column in SQLite in order to provide social media preview cards, see Social media cards for my TILs.

My favourite example of this pattern in a site that I haven't worked on myself is Mozilla.org.

They started using SQLite back in 2018 in a system they call Bedrock - Paul McLanahan provides a detailed description of how this works.

Their site content lives in a ~22MB SQLite database file, which is built and uploaded to S3 and then downloaded on a regular basis to each of their application servers.

You can view their healthcheck page to see when the database was last downloaded, and grab a copy of the SQLite file yourself. It's fun to explore that using Datasette:

Compared to static site generators

Static site generators have exploded in popularity over the past ten years. They drive the cost of hosting a site down to almost nothing, provide excellent performance, work well with CDNs and produce sites that are extremely unlikely to break.

Used carefully, the Baked Data keeps most of these characteristics while still enabling server-side code execution.

My example sites use this in a few different ways:

datasette.io provides search across 1,588 different pieces of content, plus simpler search on the plugins and tools pages. My TIL site also provides search, as does Niche Museums. All three sites provide Atom feeds that are configured using a server-side SQL query: Datasette, Niche Museums, TILs. Niche Museums offers a "Use my location" button which then serves museums near you, using a SQL query that makes use of the datasette-haversine plugin.

A common complaint about static site generators when used for larger sites is that build times can get pretty long if the builder has to generate tens of thousands of pages.

With Baked Data, 10,000 pages can be generated by a single template file and 10,000 rows in a SQLite database table.

This also makes for a faster iteration cycle during development: you can edit a template and hit "refresh" to see any page rendered by the new template instantly, without needing to rebuild any pages.

Want to give this a go?

If you want to give the Baked Data pattern a try, I recommend starting out using the combination of Datasette, GitHub Actions and Vercel. Hopefully the examples I've provided above are a good starting point - also feel free to reach out to me on Twitter or in the Datasette Discussions forum with any questions.


John Philpin : Lifestream

”Media is a word that has come to mean bad journalism.”

”Media is a word that has come to mean bad journalism.” Graham Greene

”Media is a word that has come to mean bad journalism.”

Graham Greene


”If people do not believe that mathematics is simple, it i

”If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is.” John von Neumann

”If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is.”

John von Neumann

Tuesday, 27. July 2021

John Philpin : Lifestream

🎵 The The Santana Waterboys Spandau Ballet K D Lang Lon

🎵 The The Santana Waterboys Spandau Ballet K D Lang Lonnie Donegan Gilbert O’Sullivan John Mayer Leadbelly Eclectic ? Yes … but that’s what yesterday’s musical selection was meant be.

🎵 The The
Santana
Waterboys
Spandau Ballet
K D Lang
Lonnie Donegan
Gilbert O’Sullivan
John Mayer
Leadbelly

Eclectic ?

Yes … but that’s what yesterday’s musical selection was meant be.


MyDigitalFootprint

Being Curious will not kill the #CDO

The last article was about being railroaded in the first 100 days, recognising that you are forced into a decision.  In this one I wanted to unpack that “data” shows that using science in an argument (say to defend against railroading) just makes the members of the team more partisan (aligned to their own confirmation bias and opinions).  As the #CDO, your job is to use data and science
The last article was about being railroaded in the first 100 days, recognising that you are forced into a decision.  In this one I wanted to unpack that “data” shows that using science in an argument (say to defend against railroading) just makes the members of the team more partisan (aligned to their own confirmation bias and opinions).  As the #CDO, your job is to use data and science, therefore in the first 100 days, with this insight, you are more likely to lose more people than win friends, lose more arguments than win and create bigger hurdles?   What I suggest, based on this work is that to overcome “proof by science” is to use curiosity to bring us together.

Image source: from a good article by Douglas Longenecker

---

Dan Kahan, a Yale behavioural economist, has spent the last decade studying whether the use of reason aggravates or reduces “partisan” beliefs. His research papers are here. His research shows that aggravation and alienation easily win, irrespective of being more liberal or conservative. The more we use our faculties for scientific thought, the more likely we are to take a strong position that aligns with our (original) political group (or thought). 

A way through this is to copy “solution journalism” which reports on ways that people and governments meaningfully respond to difficult problems and not on what the data says the problem is. Rather than use our best insights, analysis and thinking to reach the version of the “truth”, we use data to find ways to agree with others opinions in our communities. We help everyone to become curious.   I use the Peak Paradox framework as an approach to remaining curious as to where we are aligned and where there is a delta. 

What happens when we use data and science in our arguments and explain what the problem is, is that the individuals will selectively credit and discredit information in patterns that reflect their commitment to certain values. They (we) (I) assimilate what they (we)(I) want. 

Kahan, in 2014, asked over 1,500 respondents whether they agreed or disagreed with the following statement: “There is solid evidence of recent global warming due mostly to human activity such as burning fossil fuels.” They collected information on individuals political beliefs, and rated their science intelligence.” The analysis found that those with the least science intelligence actually have less partisan positions than those with the most. A Conservative with strong science intelligence will use their skills to find evidence against human-caused global warming, while a Liberal will find evidence for it (cognitive bias.) 

In the chart above, the y-axis represents the probability of a person agreeing that human activity caused climate change, and x-axis represents the percentile a person scored on the scientific knowledge test. The width of the bars shows the confidence interval for that probability.

As a CDO a most disconcerting finding is that individuals with more “scientific intelligence” are the quickest to align themselves on subjects they don’t know anything about. In one experiment, Kahan analyzed how people’s opinions on an unfamiliar subject is affected when given some basic scientific information, along with details about what people in their self-identified political group tend to believe about that subject. It turned out that those with the strongest scientific reasoning skills were the ones most likely to use the information to develop partisan opinions.

Critically Kahan’s research shows that people that score well on a measure called “scientific curiosity” actually show less partisanship, and it is this aspect we need to use.

As CDO’s, we need to move away from “truth”, “facts”, “data” and “right decisions” if we want to have a board and senior team who can become aligned.  We need to present ideas, concepts, how others are finding solutions and make our teams more curious.  Being curious appears to be the best way to bring us together - however counterintuitive that is.


-----

Whilst our ongoing agile iteration into information beings is never-ending, there are the first 100 days in the new role. But what to focus on? Well, that rose-tinted period of conflicting priorities is what </Hello, CDO!> is all about. Maintaining sanity when all else has been lost to untested data assumptions is a different problem entirely.



Questions to help frame your own paradoxes!

Questions to help frame your own paradoxes Leadership must be able to recognise the paradoxes created as they decide on “what they are optimising for.” The last article described two different starting points for the Peak Paradox model; finding paradox and living with paradox.  It is evident that the compromises we elect to live with become more focused as we agreed or decide on what we ar
Questions to help frame your own paradoxes

Leadership must be able to recognise the paradoxes created as they decide on “what they are optimising for.” The last article described two different starting points for the Peak Paradox model; finding paradox and living with paradox

It is evident that the compromises we elect to live with become more focused as we agreed or decide on what we are optimising for.  Such a focus has a benefit insomuch that the narrower and more articulate our view of optimised becomes, the more decisions can become aligned. However, the counter is that whilst a sharp focus and alignment will require less compromise for some, but it will equally increase the paradoxes/ compromises and tensions others will have to live with.  One team can be aligned to a vision but not necessarily on how to get there or live with the chosen “optimised” approach.  Stress is created by these differences and can create cracks and weaknesses in the team and culture. One vision, one culture, one team is very naive in any team more extensive than one.  

Sufficient dwell or thinking time is often not afforded to executive and leadership teams. Indeed board agendas are so packed and time so restricted there is little opportunity to debate or descent. The consequence of this reality is that insufficient time is allowed to consider the impacts of your optimising choices on others.  Therefore over the next few posts, I will offer questions at different levels as per the decision model (below.)  This is a link to a longer article on choices, decisions, and judgment as it frames; “are we asking the right questions?”   

The outer yellow ring is the focus of this article. Below are five questions per “Peak” that should help facilitate a conversation to maximise benefits within this space or improve understanding of the compromises other will have to make to come to your peak. 



As a reminder of what the peaks represent.  The original detailed definitions of each peak are here. 

Peak Individual Purpose.   At the exclusion of anything else, you are only interested in yourself.  Selflessness at the extreme.  You believe that you are sovereign (not having to ask anyone for permission or forgiveness), your voice and everything you say matters and everyone should agree. You are all-powerful and can do whatever you want and have the freedom and agency to do it.  

Peak Work Purpose.  At the exclusion of anything else, the only reason to work is to deliver as much possible value to the shareholders.  Employees, customers, the environment does not matter in terms of exploitation.  The purpose is to be the biggest and most efficient beast on the planet and able to deliver enormous returns; Shareholder Primacy at its most pure. Simple, clear, no conflict and no compromise, even to the point that rewarding the staff beyond fair would be a compromise.  Compliance is met with the minimal standard ensuring that nothing is wasted.  (note: it is not the reason an individual works, but a company works)

Peak Society Purpose.  At the exclusion of anything else, we have to deliver and ensure there is no suffering and poverty for any living thing. Humans must have equal education, health and safety.  There must be total transparency and equality.  Everything is equally shared, and on-one has more power, agency or influence than anyone else.  

Peak Human Purpose.  At the exclusion of anything else, we are here to escape death which we do by reproducing as much as we can with the broadest community we can.  We also have to adapt as fast as possible.  We have to meet our chemistry requirements to stay alive for as long as possible to adopt and reproduce at the expense of anything else.  Whilst all the peak purposes might be controversial (even to myself), saying purity of human purpose in chemistry/ biology might not go down very well. However, this is a model for framing thinking, so please go with it as it needs to be pure, and every other human purpose has conflicts with someone.

How would you answer these questions?

It should be evident that these questions try to frame you towards a natural one peak, but even at this level, you will have noticed that whilst you will align more to one peak than the others, but that you are full of paradoxes.  Ask different questions, you will align to a different north star. 





Ben Werdmüller

I’m a Parkland Shooting Survivor. QAnon Convinced My Dad It Was All a Hoax.

““It started a couple months into the pandemic with the whole anti-lockdown protests,” Bill said. “His feelings were so strong it turned into facts for him. So if he didn't like having to wear masks it wouldn't matter what doctors or scientists said. Anything that contradicted his feelings was wrong. So he turned to the internet to find like-minded people which led him to QAno

““It started a couple months into the pandemic with the whole anti-lockdown protests,” Bill said. “His feelings were so strong it turned into facts for him. So if he didn't like having to wear masks it wouldn't matter what doctors or scientists said. Anything that contradicted his feelings was wrong. So he turned to the internet to find like-minded people which led him to QAnon.””

[Link]


Jon Udell

Working with Postgres types

In episode 2 of this series I noted that the languages in which I’m writing Postgres functions share a common type system. It took me a while to understand how types work in the context of Postgres functions that can return sets of records and can interact with tables and materialized views. Here is a … Continue reading Working with Postgres types

In episode 2 of this series I noted that the languages in which I’m writing Postgres functions share a common type system. It took me a while to understand how types work in the context of Postgres functions that can return sets of records and can interact with tables and materialized views.

Here is a set-returning function.

create function notes_for_user_in_group( _userid text, _groupid text) returns setof annotation as $$ begin return query select * from annotation where userid = concat('acct:', _userid) and groupid = _groupid; end; $$ language plpgsql;

In this case the type that governs the returned set has already been defined: it’s the schema for the annotation table.

Column Type id uuid created timestamp without time zone updated timestamp without time zone userid text groupid text text text tags text[] shared boolean target_uri text target_uri_normalized text target_selectors jsonb references uuid[] extra jsonb text_rendered text document_id integer deleted boolean

The function returns records matching a userid and groupid. I can now find the URLs of documents most recently annotated by me.

select target_uri from notes_for_user_in_group('judell@hypothes.is', '__world__') order by created desc limit 3;

The Postgres response:

target_uri --------------------------------------------- https://news.ycombinator.com/item?id=20020501 https://www.infoworld.com/article/2886828/github-for-the-rest-of-us.html https://web.hypothes.is/help/formatting-annotations-with-markdown/ http://example.com (3 rows)

You might wonder why the function’s parameters are prefixed with underscores. That’s because variables used in functions can conflict with names of columns in tables. Since none of our column names begin with underscore, it’s a handy differentiator. Suppose the function’s signature were instead:

create function notes_for_user_in_group( userid text, groupid text)

Postgres would complain about a confict:

ERROR: column reference "userid" is ambiguous LINE 2: where userid = concat('acct:', _userid) ^ DETAIL: It could refer to either a PL/pgSQL variable or a table column.

The table has userid and groupid columns that conflict with their eponymous variables. So for functions that combine variables and database values I prefix variable names with underscore.

Set-returning functions can be called in any SQL SELECT context. In the example above that context is psql, Postgres’ powerful and multi-talented REPL (read-eval-print loop). For an example of a different context, let’s cache the function’s result set in a materialized view.

create materialized view public_notes_for_judell as ( select * from notes_for_user_in_group('judell@hypothes.is', '__world__') order by created desc ) with data;

Postgres reports success by showing the new view’s record count.

SELECT 3972

The view’s type is implicitly annotation; its schema matches the one shown above; selecting target_uri from the view is equivalent to selecting target_uri from the setof annotation returned from the function notes_for_user_in_group.

select target_uri from public_notes_for_judell limit 3;

The Postgres response is the same as above.

target_uri --------------------------------------------- https://news.ycombinator.com/item?id=20020501 https://www.infoworld.com/article/2886828/github-for-the-rest-of-us.html https://web.hypothes.is/help/formatting-annotations-with-markdown/ http://example.com (3 rows)

It shows up a lot faster though! Every time you select the function’s result set, the wrapped query has to run. For this particular example that can take a few seconds. It costs the same amount of time to create the view. But once that’s done you can select its contents in milliseconds.

Now let’s define a function that refines notes_for_user_in_group by reporting the count of notes for each annotated document.

create function annotated_docs_for_user_in_group( _userid text, _groupid text) returns table ( count bigint, userid text, groupid text, url text ) as $$ begin return query select count(n.*) as anno_count, n.userid, n.groupid, n.target_uri from notes_for_user_in_group(_userid, _groupid) n group by n.userid, n.groupid, n.target_uri order by anno_count desc; end; $$ language plpgsql;

Instead of returning a setof some named type, this function returns an anonymous table. I’ve aliased the set-returning function call notes_for_user_in_group as n and used the alias to qualify the names of selected columns. That avoids another naming conflict. If you write userid instead of n.userid in the body of the function and then call it, Postgres again complains about a conflict.

ERROR: column reference "userid" is ambiguous LINE 3: userid, ^ DETAIL: It could refer to either a PL/pgSQL variable or a table column.

Here’s a sample call to our new function..

select * from annotated_docs_for_user_in_group( 'judell', 'hypothes.is', '__world__' );

The result:

count | userid | groupid | target_uri -------+--------------------------------------------- 516 | judell@hypothes.is | __world__ | http://shakespeare.mit.edu/macbeth/full.html 73 | judell@hypothes.is | __world__ | https://www.independent.co.uk/news/world/asia/india-floods-bangladesh-nepal-deaths-millions-homeless-latest-news-updates-a7919006.html 51 | judell@hypothes.is | __world__ | https://www.usatoday.com/story/news/nation-now/2017/06/16/coconut-oil-isnt-healthy-its-never-been-healthy/402719001/

Now let’s create a view based on that function.

create materialized view url_counts_for_public_notes_by_judell as ( select * from annotated_docs_for_user_in_group( 'judell@hypothes.is', '__world__' ) ) with data;

Postgres says:

SELECT 1710

When you ask for the definition of that view using the \d command in psql:

\d url_counts_for_public_notes_by_judell

It responds with the same table definition used when creating the function.

Column | Type ---------+-------- count | bigint userid | text groupid | text url | text

Behind the scenes Postgres has created this definition from the anonymous table returned by the function.

To revise the function so that it uses a named type, first create the type.

create type annotated_docs_for_user_in_group as ( count bigint, userid text, groupid text, url text );

Postgres reports success:

CREATE TYPE

Now we can use that named type in the function. Since we’re redefining the function, first drop it.

drop function annotated_docs_for_user_in_group;

Uh oh. Postgres is unhappy about that.

ERROR: cannot drop function annotated_docs_for_user_in_group(text,text) because other objects depend on it DETAIL: materialized view url_counts_for_public_notes_by_judell depends on function annotated_docs_for_user_in_group(text,text) HINT: Use DROP ... CASCADE to drop the dependent objects too.

A view that depends on a function must be recreated when the function’s signature changes. I’ll say more about this in a future episode on set-returning functions that dynamically cache their results in materialized views. For now, since the view we just created is a contrived throwaway, just drop it along with the function by using CASCADE as Postgres recommends.

drop function annotated_docs_for_user_in_group cascade;

Postgres says:

NOTICE: drop cascades to materialized view url_counts_for_public_notes_by_judell DROP FUNCTION

Now we can recreate a version of the function that returns setof annotated_docs_for_user_in_group instead of an anonymous table(...)

create function annotated_docs_for_user_in_group( _userid text, _groupid text) returns setof annotated_docs_for_user_in_group as $$ begin return query select count(n.*) as anno_count, n.userid, n.groupid, n.target_uri from notes_for_user_in_group(_userid, _groupid) n group by n.userid, n.groupid, n.target_uri order by anno_count desc; end; $$ language plpgsql;

The results are the same as above. So why do it this way? In many cases I don’t. It’s extra overhead to declare a type. And just as a view can depend on a function, a function can depend on a type. To see why you might not want such dependencies, suppose we want to also track the most recent note for each URL.

create type annotated_docs_for_user_in_group as ( count bigint, userid text, groupid text, url text, most_recent_note timestamp );

That won’t work.

ERROR: type "annotated_docs_for_user_in_group" already exists

Dropping the type won’t work either.

ERROR: cannot drop type annotated_docs_for_user_in_group because other objects depend on it DETAIL: function annotated_docs_for_user_in_group(text,text,text) depends on type annotated_docs_for_user_in_group HINT: Use DROP ... CASCADE to drop the dependent objects too.

To redefine the type you have to do a cascading drop and then recreate functions that depend on the type. If any of those views depend on dropped functions, the drop cascades to them as well and they also must be recreated. That’s why I often write functions that return table(...) rather than setof TYPE. In dynamic languages it’s convenient to work with untyped bags of values; I find the same to be true when writing functions in Postgres.

Sometimes, though, it’s useful to declare and use types. In my experience so far it makes most sense to do that in Postgres when you find yourself writing the same returns table(...) statement in several related functions. Let’s say we want a function that combines the results of annotated_docs_for_user_in_group for some set of users.

create function annotated_docs_for_users_in_group(_userids text[], _groupid text) returns setof annotated_docs_for_user_in_group as $$ begin return query with userids as ( select unnest(_userids) as userid ) select a.* from userids u join annotated_docs_for_user_in_group(u.userid, _groupid) a on a.userid = concat('acct:', u.userid); end; $$ language plpgsql;

This new function uses the SQL WITH clause to create a common table expression (CTE) that converts an inbound array of userids into a transient table-like object, named userids, with one userid per row. The new function’s wrapped SQL then joins that CTE to the set returned from annotated_docs_for_user_in_group and returns the joined result.

(You can alternatively do this in a more procedural way by creating a loop variable and marching through the array to accumulate results. Early on I used that approach but in the context of Postgres functions I’ve come to prefer the more purely SQL-like set-oriented style.)

Sharing a common type between the two functions makes them simpler to write and easier to read. More importantly it connects them to one another and to all views derived from them. If I do decide to add most_recent_note to the type, Postgres will require me to adjust all depending functions and views so things remain consistent. That can be a crucial guarantee, and as we’ll see in a future episode it’s a key enabler of an advanced caching mechanism.

1 https://blog.jonudell.net/2021/07/21/a-virtuous-cycle-for-analytics/
2 https://blog.jonudell.net/2021/07/24/pl-pgsql-versus-pl-python-heres-why-im-using-both-to-write-postgres-functions/
3 https://blog.jonudell.net/2021/07/27/working-with-postgres-types/


John Philpin : Lifestream

‘Gibson’ … the record label …

‘Gibson’ … the record label …

‘Gibson’ … the record label …



Soldiers reporting for duty.

Soldiers reporting for duty.

Soldiers reporting for duty.

Monday, 26. July 2021

blog.deanland.com

deanland, the blog, reaches Maturity

It was over 20 years ago that deanland, the blog, first began. It ran on different blogware, actually a few different ones before settling in this Drupal version where it's been for the past (roughly) ten years. Prior to that it had been on Manila (by Userland) which I miss more and more every day. I became a wizard with Manila. It's how I taught myself HTML without knowing that's what I was

It was over 20 years ago that deanland, the blog, first began. It ran on different blogware, actually a few different ones before settling in this Drupal version where it's been for the past (roughly) ten years.

Prior to that it had been on Manila (by Userland) which I miss more and more every day. I became a wizard with Manila. It's how I taught myself HTML without knowing that's what I was doing.  

read more


Phil Windley's Technometria

Ten Reasons to Use Picos for Your Next Decentralized Programming Project

Summary: Picos are a programming model for building decentralized applications that provide significant benefits in the form of abstractions that reduce programmer effort. Here are ten eleven reasons you should use picos for your next decentralized application. Temperature Sensor Network Built from Picos I didn't start out to write a programming language that naturally supports

Summary: Picos are a programming model for building decentralized applications that provide significant benefits in the form of abstractions that reduce programmer effort. Here are ten eleven reasons you should use picos for your next decentralized application.

Temperature Sensor Network Built from Picos

I didn't start out to write a programming language that naturally supports decentralized programming using the actor-model, is cloud-native, serverless, and databaseless. Indeed, if I had, I likely wouldn't have succeeded. Instead picos evolved from a simple rule language for modifying web pages to a powerful, general-purpose programming system for building any decentralized application. This post explains what picos are and why they are a great way to build decentralized systems.

Picos are persistent compute objects. Persistence is a core feature that distinguishes picos from other programming models. Picos exhibit persistence in three ways:

Persistent identity—Picos exist, with a single identity, continuously from the moment of their creation until they are destroyed. Persistent state—Picos have persistent state that programs running in the pico can see and alter. The state is is isolated and only available inside the pico. Persistent availability—Once a pico is created, it is always on and ready to process queries and events.

Persistent identity, state, and availability make picos ideal for modeling entities of all sorts. Applications are formed from cooperating networks of picos, creating systems that better match programmer's mental models. Picos employ the actor model abstraction for distributed computation. Specifically, in response to a received message, a pico may

send messages to other picos—Picos respond to events and queries by running rules. Depending on the rules installed, a pico may raise events for itself or other picos. create other picos—Picos can create child picos and delete them. change its internal state (which can affect their behavior when the next message is received)—Each pico has a set of persistent variables that can only be affected by rules that run in response to events.

In addition to the parent-child hierarchy, picos can be arranged in a heterachical network for peer-to-peer communication and computation. A cooperating network of picos reacts to messages, changes state, and sends messages. Picos have an internal event bus for distributing those messages to rules installed in the pico. Rules in the pico are selected to run based on declarative event expressions. The pico matches events on the bus with event scenarios declared in each rule's event expressions. Any rule whose event expression matches is scheduled for execution. Executing rules may raise additional events. More detail about the event loop and pico execution model are available elsewhere.

Here are ten reasons picos are a great development environment for building decentralized applications:

Picos can be a computational node that represents or models anything: person, place, organization, smart thing, dumb thing, concept, even a pothole. Because picos encapsulate identity and state, they can be used to easily model entities of all types. Picos use a substitutable hosting model. Picos are hosted on the open-source pico engine. Picos can be moved from pico engine to pico engine without loss of functionality or a change in their identifying features1. More importantly, an application built using picos can employ picos running on multiple engines without programmer effort. Pico-based applications are scalable. Picos provide a decentralized programming model which ensures that an application can use whatever number of picos it needs without locks or multi-threading. Pico-based applications are fully sharded, meaning there is a computational node with isolated state for every entity of interest. Because these nodes can run on different engines without loss of functionality or programmer effort, pico-based applications scale fluidly. Picos provide an architectural model for trillion-node networks where literally everything is online. This provides a better, more scalable model for IoT than the current CompuServe of Things. Picos can provide high availability in spite of potentially unreliable hosting. Multiple picos, stored in many places can be used to represent a specific entity. Picos modeling a popular web page, for example, could be replicated countless times to ensure the web page is available when needed with low latency. Copies would be eventually consistent with one another and no backups would be needed. Picos are naturally concurrent without the need for locks. Each pico is an island of determinism that can be used as a building block for non-determinant decentralized systems. Each pico is isolated from any other pico, asynchronous processing is the default, and facts about the pico are published through protocols. State changes are determined by rules and respond to incoming messages seen as events. This minimizes contention and supports the efficient use of resources. Picos provide a cloud-native (internet-first) architecture that matches the application architecture to the cloud model. The pico programming model lets developers focus on business logic, not infrastructure. Applications built with picos don't merely reside in the cloud. They are architected to be performant while decentralized. Picos support reactive programming patterns that ensure applications are decoupled in both space and time to the extent possible. Picos enable stateful, databaseless programming. Picos model domain objects in code, not in database tables. Developers don't waste time setting up, configuring, or maintaining a database. Each ruleset forms a closure over the set of persistent variables used in it. Programmers simply use a variable and it is automatically persisted and available whenever a rule or function in the ruleset is executed. Picos use an extensible service model where new functionality can be layered on. Functionality within a pico is provided by rules that respond to events. An event-based programming model ensures services inside a pico are loosely coupled. And isolation of state changes between services (implemented as rulesets) inside the pico ensure a new service can be added without interfering with existing services. An event can select new rules while also selecting existing rules without the programmer changing the control flow2. Picos naturally support a Reactive programming model. Reactive programming with picos directly addresses the challenges of building decentralized applications through abstractions, programming models, data handling, protocols, interaction schemes, and error handling3. In a pico application, distribution is first class and decentralization is natural. You can read more about this in Building Decentralized Applications with Pico Networks and Reactive Programming Patterns: Examples from Fuse. Picos provide better control over terms, apps, and data. This is a natural result of the pico model where each thing has a closure over services and data. Picos cleanly separate the data for different entities. Picos, representing a specific entity, and microservices, representing a specific business capability within the pico, provide fine grained control over data and its processing. For example, if you sell your car, you can transfer the vehicle pico to the new owner, after deleting the trip service, and its associated data, while leaving untouched the maintenance records, which are stored as part of the maintenance service in the pico.

And a bonus reason for using picos:

Picos provide a better model for building the Internet of Things. Picos are an antidote to the CompuServe of Things because they provide a scalable, decentralized model for connecting everything. We built a connected car platform called Fuse to prove this model works (read more about Fuse). Picos are a natural building block for the self-sovereign internet of things (SSIoT) and can easily model necessary IoT relationship. Picos create an IoT architecture that allows interoperable interactions between devices from different manufacturers. Use Picos

If you're intrigued and want to get started with picos, there's a Quickstart along with a series of lessons. If you need help, contact me and we'll get you added to the Picolabs Slack. We'd love to help you use picos for your next distributed application.

If you're intrigued by the pico engine, the pico engine is an open source project licensed under a liberal MIT license. You can see current issues for the pico engine here. Details about contributing to the engine are in the repository's README.

Notes The caveat on this statement is that pico engines currently use URLs to identify channels used for inter-pico communication. Moving to a different engine could change the URLs that identify channels because the domain name of the engine changes. These changes can be automated. Future developments on the roadmap will reduce the use of domain names in the pico engine to make moving picos from engine to engine even easier. Note that while rules within a ruleset are guaranteed to execute in the order they appear, rules selected from different rulesets for the same event offer no ordering guarantee. When ordering is necessary, this can be done using rule chaining, guard rules, and idempotence. I first heard this expressed by Jonas Bonér in his Reactive Summit 2020 Keynote.

Tags: picos iot programming decentralization fuse ssiot


Hyperonomy Digital Identity Lab

Bootstrapping a VDR-based Fully Decentralized Object (FDO)/Credential Platform: VON Example

Michael Herman (Trusted Digital Web) 8:35 PMWhat are the common/known strategies for bootstrapping a VDR-based decentralized credential/object platform? …asked naively on purpose. Strategies for placing the first/initial DIDs in the VDR?  …presumably purposed to be the initial Issuer(s) of verifiable … Continue reading →

Michael Herman (Trusted Digital Web) 8:35 PM
What are the common/known strategies for bootstrapping a VDR-based decentralized credential/object platform? …asked naively on purpose.

Strategies for placing the first/initial DIDs in the VDR?  …presumably purposed to be the initial Issuer(s) of verifiable identifiers on the platform?

Best regards,
Michael Herman
Far Left Self-Sovereignist

Stephen Curran 5:37 PM
In Hyperledger Indy, which is a permissioned public network, the first transactions are a DID for one of the  “SuperUser’s (aka “Trustee”) of the network, and DIDs for the initial node operators that verify the transactions.  From there, DIDs for additional nodes are added, DIDs for other Trustees and then DIDs of other types of users (Endorsers, authors), who in turn create other DIDs and object types. 
If you look at von-network (https://github.com/bcgov/von-network) you can spin up a little network (4 nodes in docker) and see the transactions that are used to start the network. In that, the seed for the Trustee DID is well known, so once you’ve started the von-network, you can control it. In a “real” network, that seed (and associated private key) would of course be protected by that first Trustee.
For Sovrin, a ceremony was video’d of all the initial Trustees and Stewards (node operators) when MainNet was started in 2017.

VON Blockchain Explorer

Reference: http://greenlight.bcovrin.vonx.io/browse/domain

Reference: http://greenlight.bcovrin.vonx.io/browse/pool

Initial DID Transactions Initial Node Transactions First SCHEMA Transaction

John Philpin : Lifestream

Word  

Word  

Word

 


I know what he means.  

I know what he means.  

I know what he means.

 


🎵Faces not on albums by the Faces.    

🎵Faces not on albums by the Faces.    

🎵Faces not on albums by the Faces.

 

 


Damien Bod

Securing ASP.NET Core Razor Pages, Web APIs with Azure B2C external and Azure AD internal identities

This article shows how to implement an ASP.NET Core Razor page to authenticate against Azure B2C and use Web APIs from a second ASP.NET Core application which are also protected using Azure B2C App registrations. Azure B2C uses the signin, signup user flow and allows identities to authenticate using an Azure AD single tenant. Two […]

This article shows how to implement an ASP.NET Core Razor page to authenticate against Azure B2C and use Web APIs from a second ASP.NET Core application which are also protected using Azure B2C App registrations. Azure B2C uses the signin, signup user flow and allows identities to authenticate using an Azure AD single tenant. Two APIs are implemented, one for users and one for administrators. Only identities from the Azure AD tenant can use the administrator API. The authorization implementation which forces this, is supported using an ASP.NET Core policy with a handler.

Code: https://github.com/damienbod/azureb2c-fed-azuread

Setup

The ASP.NET Core applications only use Azure B2C to authenticate and authorize. An ASP.NET Core Razor page application is used for the UI, but this can be any SPA, Blazor app or whatever the preferred tech stack is. The APIs are implemented using ASP.NET Core and this uses Azure B2C to validate and authorize the access tokens. The application accepts two different access tokens from the same Azure B2C identity provider. Each API has a separate scope from the associating Azure App registration. The Admin API uses claims specific to the Azure AD identity to authorize only Azure AD internal users. Other identities cannot use this API and this needs to be validated. The Azure B2C identity provider federates to the Azure AD single tenant App registration.

Setup Azure B2C App registrations

Three Azure App registrations were created in Azure B2C to support the setup above. Two for the APIs and one for the UI. The API Azure App registrations are standard with just a scope definition. The scope access_as_user was exposed in both and the APIs can be used for user access.

The UI Azure App registration is setup to use an Azure B2C user flow and will have access to both APIs. You need to select the options with the user flows.

Add the APIs to the permissions of the Azure app registration for the UI application.

Setup Azure AD App registration

A single tenant Azure App registration needs to be created in the Azure AD for the internal or admin users. The redirect URL for this is https://”your-tenant-specific-path”/oauth2/authresp. This will be used from an Azure B2C social login using the Open ID Connect provider. You also need to define a user secret and use this later. At present only secrets can be defined in this UI. This is problematic because the secrets have a max expiry of 2 years, if defining this in the portal.

Setup Azure B2C identity provider

A custom identity provider needs to be created to access the single tenant Azure AD for the admin identity authentication. Select the Identity providers in Azure B2C and create a new Open ID Connect custom IDP. Add the data to match the Azure App registration created in the previous step.

Setup Azure B2C user flow

Now a signin, signup user flow can be created to implement the Azure B2C authentication process and the registration process. This will allow local Azure B2C guest users and also the internal administrator users from the Azure AD tenant. The idp claim is required and idp_access_token claim if you require user data from the Azure AD identity. Add the required claims when creating the user flow. The claims can be added when creating the user flow in the User attributes and token claims section. Select the custom Open ID Connect provider and add this to the flow as well.

The user flow is now setup. The Azure App registrations can now be used to login and use either API as required. The idp and the idp_access_token are added for the Azure AD sign-in and this can be validated when using the admin API.

Implementing ASP.NET Core Razor page with Microsoft.Identity.Web

The ASP.NET Core application is secured using the Microsoft.Identity.Web and the Microsoft.Identity.Web.UI Nuget packages. These packages implement the Open ID connect clients and handles the Azure B2C specific client handling. The AddMicrosoftIdentityWebAppAuthentication method is used to add this and the AzureAdB2C configuration is defined to read the configuration from the app.settings, user secrets, key vault or whatever deployment is used. The rest is standard ASP.NET Core setup.

public void ConfigureServices(IServiceCollection services) { services.AddTransient<AdminApiOneService>(); services.AddTransient<UserApiOneService>(); services.AddHttpClient(); services.AddOptions(); string[] initialScopes = Configuration.GetValue<string>("UserApiOne:ScopeForAccessToken")?.Split(' '); services.AddMicrosoftIdentityWebAppAuthentication(Configuration, "AzureAdB2C") .EnableTokenAcquisitionToCallDownstreamApi(initialScopes) .AddInMemoryTokenCaches(); services.AddRazorPages().AddMvcOptions(options => { var policy = new AuthorizationPolicyBuilder() .RequireAuthenticatedUser() .Build(); options.Filters.Add(new AuthorizeFilter(policy)); }).AddMicrosoftIdentityUI(); } public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); } else { app.UseExceptionHandler("/Error"); app.UseHsts(); } app.UseHttpsRedirection(); app.UseStaticFiles(); app.UseRouting(); app.UseAuthentication(); app.UseAuthorization(); app.UseEndpoints(endpoints => { endpoints.MapRazorPages(); endpoints.MapControllers(); }); }

The Micorosoft.Identity.Web package uses the AzureAdB2C settings for the configuration. This example is using Azure B2C and the configuration for Azure B2C is different to an Azure AD configuration. The Instance MUST be set to domain of the Azure B2C tenant and the SignUpSignInPolicyId must be set to use the user flow as required. A signin, signup user flow is used here. The rest is common for be Azure AD and Azure B2C settings. The ScopeForAccessToken matches the two Azure App registrations created for the APIs.

"AzureAdB2C": { "Instance": "https://b2cdamienbod.b2clogin.com", "ClientId": "8cbb1bd3-c190-42d7-b44e-42b20499a8a1", "Domain": "b2cdamienbod.onmicrosoft.com", "SignUpSignInPolicyId": "B2C_1_signup_signin", "TenantId": "f611d805-cf72-446f-9a7f-68f2746e4724", "CallbackPath": "/signin-oidc", "SignedOutCallbackPath ": "/signout-callback-oidc" }, "UserApiOne": { "ScopeForAccessToken": "https://b2cdamienbod.onmicrosoft.com/723191f4-427e-4f77-93a8-0a62dac4e080/access_as_user", "ApiBaseAddress": "https://localhost:44395" }, "AdminApiOne": { "ScopeForAccessToken": "https://b2cdamienbod.onmicrosoft.com/5f4e8bb1-3f4e-4fc6-b03c-12169e192cd7/access_as_user", "ApiBaseAddress": "https://localhost:44395" },

The Admin Razor page uses the AuthorizeForScopes to authorize for the API it uses. This Razor page uses the API service to access the admin API. No authorization is implemented in the UI to validate the identity. Normally the page would be hidden if the identity is not an administrator , I left this out in so that it is easier to validate this in the API as this is only a demo.

namespace AzureB2CUI.Pages { [AuthorizeForScopes(Scopes = new string[] { "https://b2cdamienbod.onmicrosoft.com/5f4e8bb1-3f4e-4fc6-b03c-12169e192cd7/access_as_user" })] public class CallAdminApiModel : PageModel { private readonly AdminApiOneService _apiService; public JArray DataFromApi { get; set; } public CallAdminApiModel(AdminApiOneService apiService) { _apiService = apiService; } public async Task OnGetAsync() { DataFromApi = await _apiService.GetApiDataAsync().ConfigureAwait(false); } } }

The API service uses the ITokenAcquisition to get an access token for the defined scope. If the identity and the Azure App registration are authorized to access the API, then an access token is returned for the identity. This is sent using a HttpClient created using the IHttpClientFactory interface.

using Microsoft.Extensions.Configuration; using Microsoft.Identity.Web; using Newtonsoft.Json.Linq; using System; using System.Net.Http; using System.Net.Http.Headers; using System.Threading.Tasks; namespace AzureB2CUI { public class AdminApiOneService { private readonly IHttpClientFactory _clientFactory; private readonly ITokenAcquisition _tokenAcquisition; private readonly IConfiguration _configuration; public AdminApiOneService(IHttpClientFactory clientFactory, ITokenAcquisition tokenAcquisition, IConfiguration configuration) { _clientFactory = clientFactory; _tokenAcquisition = tokenAcquisition; _configuration = configuration; } public async Task<JArray> GetApiDataAsync() { var client = _clientFactory.CreateClient(); var scope = _configuration["AdminApiOne:ScopeForAccessToken"]; var accessToken = await _tokenAcquisition.GetAccessTokenForUserAsync(new[] { scope }).ConfigureAwait(false); client.BaseAddress = new Uri(_configuration["AdminApiOne:ApiBaseAddress"]); client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken); client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json")); var response = await client.GetAsync("adminaccess").ConfigureAwait(false); if (response.IsSuccessStatusCode) { var responseContent = await response.Content.ReadAsStringAsync().ConfigureAwait(false); var data = JArray.Parse(responseContent); return data; } throw new ApplicationException($"Status code: {response.StatusCode}, Error: {response.ReasonPhrase}"); } } }

Implementing the APIs with Microsoft.Identity.Web

The ASP.NET Core project implements the two separate APIs using separate authentication schemes and policies. The AddMicrosoftIdentityWebApiAuthentication method configures services for the user API using the default JWT scheme “Bearer” and the second scheme is setup for the “BearerAdmin” JWT bearer auth for the admin API. All API calls require an authenticated user which is setup in the AddControllers using a global policy. The AddAuthorization method is used to add an authorization policy for the admin API. The IsAdminHandler handler is used to fulfil the IsAdminRequirement requirement.

public void ConfigureServices(IServiceCollection services) { services.AddHttpClient(); services.AddOptions(); JwtSecurityTokenHandler.DefaultInboundClaimTypeMap.Clear(); // IdentityModelEventSource.ShowPII = true; services.AddMicrosoftIdentityWebApiAuthentication( Configuration, "AzureB2CUserApi"); services.AddMicrosoftIdentityWebApiAuthentication( Configuration, "AzureB2CAdminApi", "BearerAdmin"); services.AddControllers(options => { var policy = new AuthorizationPolicyBuilder() .RequireAuthenticatedUser() // disabled this to test with users that have no email (no license added) // .RequireClaim("email") .Build(); options.Filters.Add(new AuthorizeFilter(policy)); }); services.AddSingleton<IAuthorizationHandler, IsAdminHandler>(); services.AddAuthorization(options => { options.AddPolicy("IsAdminRequirementPolicy", policyIsAdminRequirement => { policyIsAdminRequirement.Requirements.Add(new IsAdminRequirement()); }); }); }

The IsAdminHandler class checks for the idp claim and validates that the single tenant we require for admin identities is used to sign-in. The access token needs to be validated that the token was issued by our Azure B2C and that it has the correct scope. Since this is done in using the Microsoft.Identity.Web attributes, we don’t need to do this here.

public class IsAdminHandler : AuthorizationHandler<IsAdminRequirement> { protected override Task HandleRequirementAsync( AuthorizationHandlerContext context, IsAdminRequirement requirement) { if (context == null) throw new ArgumentNullException(nameof(context)); if (requirement == null) throw new ArgumentNullException(nameof(requirement)); var claimIdentityprovider = context.User.Claims.FirstOrDefault(t => t.Type == "idp"); // check that our tenant was used to signin if (claimIdentityprovider != null && claimIdentityprovider.Value == "https://login.microsoftonline.com/7ff95b15-dc21-4ba6-bc92-824856578fc1/v2.0") { context.Succeed(requirement); } return Task.CompletedTask; } }

The AdminAccessController class is used to provide the admin data for admin identities. The BearerAdmin scheme is required and the IsAdminRequirementPolicy policy. The access token admin scope is also validated.

[Authorize(AuthenticationSchemes = "BearerAdmin", Policy = "IsAdminRequirementPolicy")] [AuthorizeForScopes(Scopes = new string[] { "api://5f4e8bb1-3f4e-4fc6-b03c-12169e192cd7/access_as_user" })] [ApiController] [Route("[controller]")] public class AdminAccessController : ControllerBase { [HttpGet] public List<string> Get() { string[] scopeRequiredByApi = new string[] { "access_as_user" }; HttpContext.VerifyUserHasAnyAcceptedScope(scopeRequiredByApi); return new List<string> { "admin data" }; } }

The user API also validates the access token, this time using the default Bearer scheme. No policy is required here, so only the default global authorization filter is used. The user API scope is validated.

[Authorize(AuthenticationSchemes = "Bearer")] [AuthorizeForScopes(Scopes = new string[] { "api://723191f4-427e-4f77-93a8-0a62dac4e080/access_as_user" })] [ApiController] [Route("[controller]")] public class UserAccessController : ControllerBase

When the application is run, the Azure B2C user flow is used to authenticate and internal or external users can sign-in, sign-up. This view can be customized to match your styles.

Admins can use the admin API and the guest users can use the user APIs.

Notes

This works but it can be improved and there are other ways to achieve this setup. If you require only a subset of identities from the Azure AD tenant, an enterprise app can be used to define the users which can use the Azure AD App registration. Or you can do this with an Azure group and assign this to the app and the users to the group.

You should also force MFA in the application for admins by validating the claims in the token and also the client ID which the token was created for. (as well as in the Azure AD tenant.)

Azure B2C is still using version one access tokens and seems and the federation to Azure AD does not use PKCE.

The Open ID Connect client requires a secret to access the Azure AD App registration. This can only be defined for a max of two years and it is not possible to use managed identities or a certificate. This means you would need to implement a secret rotation script or something so that to solution does not stop working. This is not ideal in Azure and is solved better in other IDPs. It should be possible to define long living secrets using the Powershell module and you update then with every release etc.

It would also be possible to use the Graph API to validate the identity accessing the admin API, user API.

Azure B2C API connectors could also be used to add extra claims to the tokens for usage in the application.

Links:

https://docs.microsoft.com/en-us/azure/active-directory-b2c/overview

https://docs.microsoft.com/en-us/azure/active-directory-b2c/identity-provider-azure-ad-single-tenant?pivots=b2c-user-flow

https://github.com/AzureAD/microsoft-identity-web

https://docs.microsoft.com/en-us/azure/active-directory/develop/microsoft-identity-web

https://docs.microsoft.com/en-us/azure/active-directory-b2c/identity-provider-local

https://docs.microsoft.com/en-us/azure/active-directory/

https://docs.microsoft.com/en-us/aspnet/core/security/authentication/azure-ad-b2c

https://github.com/azure-ad-b2c/azureadb2ccommunity.io

https://github.com/azure-ad-b2c/samples

Sunday, 25. July 2021

Ben Werdmüller

Every single time I’m in Portland - ...

Every single time I’m in Portland - including just overnight, like right now - I don’t want to leave. There might be a solution for this. I wish I had more time to see people!

Every single time I’m in Portland - including just overnight, like right now - I don’t want to leave. There might be a solution for this.

I wish I had more time to see people!


Here's Tom with the Weather

At the cloak room

At the cloak room


John Philpin : Lifestream

Pity - I was looking for the Bad Hemp.  

Pity - I was looking for the Bad Hemp.  

Pity - I was looking for the Bad Hemp.

 

Saturday, 24. July 2021

Simon Willison

How the Python import system works

How the Python import system works Remarkably detailed and thorough dissection of how exactly import, modules and packages work in Python - eventually digging right down into the C code. Part of Victor Skvortsov's excellent "Python behind the scenes" series. Via Hacker News

How the Python import system works

Remarkably detailed and thorough dissection of how exactly import, modules and packages work in Python - eventually digging right down into the C code. Part of Victor Skvortsov's excellent "Python behind the scenes" series.

Via Hacker News


Jon Udell

pl/pgsql versus pl/python? Here’s why I’m using both to write Postgres functions.

In A virtuous cycle for analytics I noted that our library of Postgres functions is written in two languages: Postgres’ built-in pl/pgsql and the installable alternative pl/python. These share a common type system and can be used interchangeably. Here’s a pl/pgsql classifier that tries to match the name of a course against a list of … Continue reading pl/pgsql versus pl/python? Here’s why I’m using

In A virtuous cycle for analytics I noted that our library of Postgres functions is written in two languages: Postgres’ built-in pl/pgsql and the installable alternative pl/python. These share a common type system and can be used interchangeably.

Here’s a pl/pgsql classifier that tries to match the name of a course against a list of patterns that characterize the humanities.

create function humanities_classifier(course_name text) returns boolean as $$ begin return lower(course_name) ~ any(array[ 'psych', 'religio', 'soci' ]); end; $$ language plpgsql; # select humanities_classifier('Religious Studies 101') as match; match ----- t # select humanities_classifier('Comparative Religions 200') as match; match ----- t

Here is that same classifier in Python.

create function humanities_classifier(course_name text) returns boolean as $$ sql = f""" select lower('{course_name}') ~ any(array[ 'psych', 'religio', 'soci' ]) as match""" results = plpy.execute(sql) return results[0]['match'] $$ language plpython3u; # select humanities_classifier('Religious Studies 101') as match; match ----- t # select humanities_classifier('Comparative Religions 200') as match; match ----- t

The results are exactly the same. In this case, Python is only wrapping the SQL used in the orginal function and interpolating course_name into it. So why use pl/python here? I wouldn’t. The pl/pgsql version is cleaner and simpler because the SQL body doesn’t need to be quoted and course_name doesn’t need to be interpolated into it.

Here’s a more Pythonic version of the classifier.

create function humanities_classifier(course_name text) returns boolean as $$ import re regexes = [ 'psych', 'religio', 'soci' ] matches = [r for r in regexes if re.search(r, course_name, re.I)] return len(matches) $$ language plpython3u;

There’s no SQL here, this is pure Python. Is there any benefit to doing things this way? In this case probably not. The native Postgres idiom for matching a string against a list of regular expressions is cleaner and simpler than the Python technique shown here. A Python programmer will be more familiar with list comprehensions than with the Postgres any and ~ operators but if you’re working in Postgres you’ll want to know about those, and use them not just in functions but in all SQL contexts.

What about performance? You might assume as I did that a pl/pgsql function is bound to be way faster than its pl/python equivalent. Let’s check that assumption. This SQL exercises both flavors of the function, which finds about 500 matches in a set of 30,000 names.

with matching_courses as ( select humanities_classifier(name) as match from lms_course_groups ) select count(*) from matching_courses where match;

Here are the results for three runs using each flavor of the function:

pl/pgsql: 159ms, 201ms, 125ms pl/python: 290ms, 255ms, 300ms

The Python flavor is slower but not order-of-magnitude slower; I’ve seen cases where a pl/python function outperforms its pl/pgsql counterpart.

So, what is special about Python functions inside Postgres? In my experience so far there are three big reasons to use it.

Python modules

The ability to wield any of Python’s built-in or loadable modules inside Postgres brings great power. That entails great responsibility, as the Python extension is “untrusted” (that’s the ‘u’ in ‘plpython3u’) and can do anything Python can do on the host system: read and write files, make network requests.

Here’s one of my favorite examples so far. Given a set of rows that count daily or weekly annotations for users in a group — so for weekly accounting each row has 52 columns — the desired result for the whole group is the element-wise sum of the rows. That’s not an easy thing in SQL but it’s trivial using numpy, and in pl/python it happens at database speed because there’s no need to transfer SQL results to an external Python program.

Metaprogramming

It’s often useful to write that functions that write and then run SQL queries. It’s overkill for simple variable interpolation; as shown above pl/pgsql does that handily without the cognitive overhead and visual clutter of poking values into a SQL string. For more advanced uses that compose queries from SQL fragments, though, pl/pgsql is hopeless. You can do that kind of thing far more easily, and more readably, in Python.

Introspection

A pl/python function can discover and use its own name. That’s the key enabler for a mechanism that memoizes the results of a function by creating a materialized view whose name combines the name of the function with the value of a parameter to the function. This technique has proven to be wildly effective.

I’ll show examples of these scenarios in later installments of this series. For now I just want to explain why I’ve found these two ways of writing Postgres functions to be usefully complementary. The key points are:

– They share a common type system.

– pl/pgsql, despite its crusty old syntax, suffices for many things.

– pl/python leverages Python’s strengths where they are most strategic

When I began this journey it wasn’t clear when you’d prefer one over the other, or why it might make sense to use both in complementary ways. This installment is what I’d like to have known when I started.


1 https://blog.jonudell.net/2021/07/21/a-virtuous-cycle-for-analytics/
2 https://blog.jonudell.net/2021/07/24/pl-pgsql-versus-pl-python-heres-why-im-using-both-to-write-postgres-functions/
3 https://blog.jonudell.net/2021/07/27/working-with-postgres-types/


John Philpin : Lifestream

Morning  

Morning  

Morning

 

Friday, 23. July 2021

Ben Werdmüller

Why I oppose recall of DA Chesa Boudin

"These reforms were desperately needed: disproportionate treatment of Black and Brown people in arrests, prosecutions, and lifetimes of incarceration isn't just a trend of a few decades — it has taken place over centuries in America. Many in our LGBTQ+ community are in desperate need of criminal justice reform now." I oppose his recall too. [Link]

"These reforms were desperately needed: disproportionate treatment of Black and Brown people in arrests, prosecutions, and lifetimes of incarceration isn't just a trend of a few decades — it has taken place over centuries in America. Many in our LGBTQ+ community are in desperate need of criminal justice reform now." I oppose his recall too.

[Link]


Fairness Friday: Southerners on New Ground

Inspired by Fred Wilson’s Funding Fridays, which highlight a new crowdfunding campaign that he’s contributed to every week, I decided to start a series of my own. This isn’t a knock on him: I genuinely enjoy those posts. But I also felt like there was room for something else. Starting this week, I’m going to be posting Fairness Fridays: a new community social justice organization each week. I w

Inspired by Fred Wilson’s Funding Fridays, which highlight a new crowdfunding campaign that he’s contributed to every week, I decided to start a series of my own. This isn’t a knock on him: I genuinely enjoy those posts. But I also felt like there was room for something else.

Starting this week, I’m going to be posting Fairness Fridays: a new community social justice organization each week. I will donate to each featured organization. If you feel so inclined, please join me.

This week, I’m donating to Southerners on New Ground. Based in Atlanta, SONG describes its mission as follows:

We build membership (or our base) as a way to build the skills, connection and leadership of thousands of Southern rural LGBTQ people of color, immigrant people and working class people – united together in the struggle for dignity and justice for all people. In order to transform the South we must build our collective power, our people power, which comes from thousands and thousands of us uniting to make the South the home our communities need it to be.

Its work includes bail reform, Black queer, trans, and gender non-conforming leadership work, and electoral justice. It’s all good stuff.

I donated, became a member, and offered to use my tech skills for community members who need them. If you have the means, I encourage you to do the same.

 

Photo by Nathania Johnson

#justice


John Philpin : Lifestream

Just added a More on my micro blog site - all built in Craft

Just added a More on my micro blog site - all built in Craft - for those thinking about how to use that nifty little ‘notes’ app.

Just added a More on my micro blog site - all built in Craft - for those thinking about how to use that nifty little ‘notes’ app.


Simon Willison

The Tyranny of Spreadsheets

The Tyranny of Spreadsheets In discussing the notorious Excel incident last year when the UK lost track of 16,000 Covid cases due to a .xls row limit, Tim Harford presents a history of the spreadsheet, dating all the way back to Francesco di Marco Datini and double-entry bookkeeping in 1396. A delightful piece of writing. Via Hacker News

The Tyranny of Spreadsheets

In discussing the notorious Excel incident last year when the UK lost track of 16,000 Covid cases due to a .xls row limit, Tim Harford presents a history of the spreadsheet, dating all the way back to Francesco di Marco Datini and double-entry bookkeeping in 1396. A delightful piece of writing.

Via Hacker News


John Philpin : Lifestream

Extra podcast thought @danielpunkass .. surely a subscriptio

Extra podcast thought @danielpunkass .. surely a subscription approach perfectly lends itself to shipping and then continuous enhancements - not ship and hope?

Extra podcast thought @danielpunkass .. surely a subscription approach perfectly lends itself to shipping and then continuous enhancements - not ship and hope?


Many people use the GTD approach … me? I am starting up ‘JGS

Many people use the GTD approach … me? I am starting up ‘JGS’ … thanks @manton and @danielpunkass

Many people use the GTD approach … me? I am starting up ‘JGS’ … thanks @manton and @danielpunkass

Thursday, 22. July 2021

Ben Werdmüller

A Defunct Video Hosting Site Is Flooding Normal Websites With Hardcore Porn

"As pointed out by Twitter user @dox_gay, hardcore porn is now embedded on the pages of the Huffington Post, New York magazine, The Washington Post, and a host of other websites. This is because a porn site called 5 Star Porn HD bought the domain for Vidme, a brief YouTube competitor founded in 2014 and shuttered in 2017. Its Twitter account is still up, but the domain lapsed.

"As pointed out by Twitter user @dox_gay, hardcore porn is now embedded on the pages of the Huffington Post, New York magazine, The Washington Post, and a host of other websites. This is because a porn site called 5 Star Porn HD bought the domain for Vidme, a brief YouTube competitor founded in 2014 and shuttered in 2017. Its Twitter account is still up, but the domain lapsed." Cool URIs don't change, etc etc.

[Link]


Simon Willison

Datasette - an ecosystem of tools for working with small data

This is the transcript and video from a talk I gave at PyGotham 2020 about using SQLite, Datasette and Dogsheep to work with small data. I really like the term "small data", as a counterpoint to big data. I define small data as anything that fits on a mobile phone, and since my iPhone has half a terabyte of storage these days that ends up covering a lot of ground! I think SQLite is the ideal t

This is the transcript and video from a talk I gave at PyGotham 2020 about using SQLite, Datasette and Dogsheep to work with small data.

I really like the term "small data", as a counterpoint to big data. I define small data as anything that fits on a mobile phone, and since my iPhone has half a terabyte of storage these days that ends up covering a lot of ground!

I think SQLite is the ideal tool for working with data that fits this definition.

My PyGotham talk demonstrates how to find SQLite databases that already exist on your laptop, how to use Datasette to explore them, and then expands to cover Datasette as a tool for publishing data online and my Dogsheep family of tools for personal analytics.

Here's the talk video on YouTube. I've included a full transcript below with additional links and embedded code samples.

Transcript and notes

I'm going to spend the next 20 minutes trying to convince you that almost every data problem you have should be solved using SQLite.

This is also an excuse for me to promote open source project I've been working on for the last three years, called Datasette.

Datasette is a tool for exploring and publishing data that's stored in SQLite databases. So the more people I can use convince to use SQLite to solve things the better!

So let's talk about SQLite, which claims to be the most widely deployed and used database in the world, and I think that claim holds up.

You may not have realised it, but you're using SQLite every single day. It's in phones and laptops. It's in embedded devices. I have an Apple Watch, so I'm wearing SQLite databases on my wrist right now.

Let's take a look on my laptop and see what we can find.

So this right here is a magic incantation that on a Mac will search for the largest SQLite databases across the entire system.

mdfind "kMDItemDisplayName == *.sqlite" -0 | \ xargs -0 stat "-f%z %N" | sort -nr | head -n 20

I'm gonna run this right now to get my top 20. And here we are. Evernote is using SQLite, Firefox uses it, but the one at the top, this one is 857 megabytes for SQLite file. It's something to do with Apple Photos, so let's see what's in there.

Datasette can be installed as a command-line application, you can run it against the path to a SQLite database, and it starts up a little local web server, which you can then use to explore that data.

So there are 67 tables in this Apple Photos database, with detected face prints and face crops and albums and all sorts of things.

But the one that I happen to know is super interesting is this one here. It's called the Z Cloud Master Media Meta Data. It has 44,000 rows in and I happen to know I've taken about 44,000 photographs - so it looks like there's some interesting stuff here.

This is kind of garbled, but there's a clue. The binary column here starts with BPlist, and I happen to know that Apple have a format called a binary plist, which is probably what this is.

So I'm gonna install a plugin for Datasette called datasette-bplist that knows what to do with that data. This is being installed from PyPI.

datasette install datasette-bplist

And now I can start Datasette up again. Hit refresh on this page. And now that Z Data has been decoded into something that's a little bit more readable.

And straightaway we can see that OK, this is classic photo metadata. We've got the make and model of the device. we've got EXIF data. The really exciting stuff is we've got GPS data. So this is a database that includes the latitude and longitude of every photograph I've ever taken, and we're absolutely going to dig into that and see if we can do some interesting stuff with it later on.

So SQLite is clearly a pretty great database - if Apple are using it this extensively it must have some good things going on for it. But then why don't we use it for web applications?

Well, the classic answer to that is that the one thing SQLite is not so good at is accepting concurrent writes. If you've got a high traffic web site with lots of people writing to your database, you're better off with something like MySQL or Postgres.

But I realised that I have a whole bunch of use cases that don't require any writes at all. Any time I want to publish data online that isn't going to change, SQLite is actually a really good fit for that.

So that was the original inspiration for Datasette. I was thinking about how if I want to publish data online, is this a way I can do it that's really cheap and flexible and interesting?

So that's what I've been doing. I'll show you a few examples.

This is Covid-19 case data, which is published online right now by Johns Hopkins University. The New York Times, the L.A. Times and The Economist.

https://covid-19.datasettes.com/

They're all publishing their data as CSV files in GitHub repositories, which is pretty great because you get a commit history of changes to that CSV data. But it's not necessarily something you can dive straight into and start using.

That's where Datasette comes in. This is a Datasette instance which is automatically updated every day with the latest data from these different publications and I can drill through and see things like this New York Times data for example.

The New York Times publishes county level data of cases and deaths from Covid-19 and they've got over half a million rows in this spreadsheet. And so when we look at this data, it's pretty unexciting in a sort of table form. We've got the date, the county, the state, the FIPS code, which is a unique identify for each county, the number of cases and the number of deaths.

But Datasette kicks in and does some really useful things for us.

Firstly, Datasette has a concept of faceting where it looks for columns in the database which have a small number of unique values - like the State column for example, which only has 50 values in it, and it makes those available for navigation at the top along with these summary counts. So I can look down at this and say, Oh, look, Tennessee. Let's drill into just the rows in this table that mention Tennessee. And then Tennessee has 16,000 rows of which each county has a whole bunch. So let's look at Rutherford County in Tennessee. That's 181 rows. And now that we've drilled down these two levels, we've got back a table that is just showing us the number of cases and number of deaths ordered my date.

So each of these rows says on the sixth of September, there have been 80 deaths reported in 8100 cases.

You saw a Datasette plugin earlier when I installed the bplist plugin. Another plugin I wrote is called Datasette Vega. It gives you simple charting options, so we can say I want a bar chart that plots the date on the X axis against the number of cases on the Y axis. And now we've got a chart of cases over time in this particular county in the state of Tennessee.

A couple of other things you can do: You can export the data back out again so I could get this data as JSON if I want to load this programmatically into something, and get it out as CSV if I want to load it into something like Microsoft Excel.

I can actually export it as YAML because there's an experimental plugin I've been building that does a YAML export. And then, more importantly, you can view and edit the SQL query that was used for this page.

Now this is one of the interesting things about publishing read-only data: because the data as read only and it's opened in a special read only mode, people executing SQL queries can't cause any damage to that database. So SQL injection which in most web applications is a serious security vulnerability for us, becomes a feature.

We could say I just want back the data on the cases and the deaths columns - I'm gonna hit "run SQL" and now I'm getting back just that data that I requested, and I could export that data back out again as JSON or CSV. So this becomes a really powerful way of remixing and slicing and dicing data and then getting it back out again in a format that I can use somewhere else.

So that's all well and good. But the obvious question is, how do you get this data into a SQLite file in the first place? So I'm gonna show you a demo using one of my favourite pieces of data of all time. This is the results of the 2018 New York Central Park squirrel census, where a bunch of volunteers went out and tried to answer the crucial question: "How many squirrels are there in Central Park?".

And this is published as a CSV file, it's got 3,023 rows. Each row is a squirrel. And we can export that out and get back CSV. So I've got the file here. I'm gonna turn it into a SQLite database.

I have a tool I wrote called csvs-to-sqlite, which takes a CSV file - or multiple CSV files - and turns them into, or loads them into SQLite. So here we go, I've run that command and and now I have squirrels.db, which is a SQLite database.

csvs-to-sqlite 2018_Central_Park_Squirrel_Census_-_Squirrel_Data.csv squirrels.db

If I run sqlite3 squirrels.db .schema, it'll show me the schema - and it is indeed a table with columns for squirrels with unique squirrel IDS and which hectare they're in and all of that kind of stuff.

datasette squirrels.db

And so now that we've done that, I can run datasette squirrels.db and start up Datasette and I can see that data in my browser. And here we go here, here are 3,000 rows of squirrel data.

Faceting is kind of fun: you can see things like primary fur colour, showing up as options there. And, actually, if we facet by that, we can say you know what? For the 392 cinnamon squirrels, let's see what the highlight fur colours are for those. Show me the 10 squirrels that are black and have cinnamon hair.

Another thing that stands out about this data is that there are these two columns here, X and Y, which look suspiciously like they might be latitudes and longitudes. We can do something really interesting with that.

I'm gonna load up the SQLite command-line tool for squirrels.db, and I'm gonna rename the X column to longitude.

sqlite3 squirrels.db alter table "2018_Central_Park_Squirrel_Census_-_Squirrel_Data" rename column X to longitude; alter table "2018_Central_Park_Squirrel_Census_-_Squirrel_Data" rename column Y to latitude;

So "alter table" - um, quite a long table name this one - gonna alter this table here. Rename column X to longitude. And I'm gonna rename column Y to latitude. And now I'm going to check my datasette plugins and make sure I've got a plugin installed called datasette-cluster-map.

There it is. So now if I run Datasette against squirrels again and refresh this page, the plugin here notices the latitude and longitude columns and it draws them on the map. So this is a map of everywhere that they saw a cinnamon squirrel with black highlight fur.

And if I close off these filters you can see that here's a map of all 3,000 squirrels that were spotted in Central Park. And I can zoom in and explore the data that way.

That's pretty cool. Wouldn't it be cool if we could share this data on the Internet somehow?

So we've got a SQLite database. And this is where another key feature of Datasette comes in, which is the"datasette publish command.

Datasette Publish lets you publish databasees online, using Google Cloud Run or Heroku or Vercel. I'm going to use Vercel right now, so if I type datasette publish vercel squirrels.db - I need to give it a project name. I'm going to call it pygotham-squirrels. And I know that I want to install that datasette-cluster-map plugin as well.

datasette publish vercel squirrels.db \ --project=pygotham-squirrels \ --install=datasette-cluster-map

So I run this command, and it packages up that database with the Datasette application itself, uploads it and starts running a build on Vercel. It gives me a URL here. I can watch it doing its thing - so it's installing runtimes, it's installing those required dependencies, that additional plugin. And normally, this only takes about 20 seconds to run. And at the end of that 20 seconds, I will have a Web application that is running online with that data.

https://pygotham-squirrels.vercel.app/

So anyone can visit this URL right now, click on that map and they get that Datasette of squirrels with the map plugin, with the ability to facet by primary fur colour. It's all up there. It's running online and it's ready for other people to visit.

I designed this initially for newspapers based on my experience working at the Guardian newspaper, where we wanted to publish some of the data behind our stories, but as a way productively getting structured data onto the web in a way you can start using it I think this is really, really exciting as a capability.

And remember there are JSON APIs for this. So we have just published an API online for squirrel data that anyone who can write JavaScript or Python can start using to build their own squirrel maps.

Another thing that I've been doing with this is figuring out how to turn these into much more custom experiences.

Datasette allows you to provide custom templates, it lets you provide custom CSS, which means you can actually use it to build entire websites powered under the hood by Datasette.

One of the first I built was this one here. This is called niche-museums.com - it's a website for my hobby of trying to track down weird and niche museums around the world, I can click "Use my location" and see just the weird museums near me.

There's one just around the corner from our house called The Comic Book Rockstars Toilet Seat Museum, where a local comic book store has been collecting toilet seats illustrated by famous comic artists. And if you're ever in San Francisco, you should absolutely pop by Hayes Valley and check this place out.

But this right here is just a heavily styled Datasette instance. If you go to /browse, it will give you access to the data. You can see that each museum is a row in a SQLite database table. I've got that plugin, so I get a free map showing you where everything is. Then the actual museum pages are just templated versions of the underlying Datasette pages.

I really like this as a way of publishing websites because it's gives you all of the benefits of static publishing where you it scales magically by starting up new instances, you don't have to worry about security holes and stuff because it's all read only data, but because there's a relational database under it, you can build features like search - so I can search for Bigfoot and get back the Bigfoot Discovery Museum.

Or you can build things like this little "use my location", button at the top of the screen. But again, this is all built using Datasette and Datasette plugins. I have a plugin that knows how to turn a SQL query into an Atom feed so that people can subscribe to new updates to the website as well.

But to finish, I want to talk about a totally different use-case for all of this stuff, and that's personal analytics.

This is an idea that I had had a year and a half ago, inspired by an essay written by Stephen Wolfram. Stephen wrote this absolutely fascinating essay about his approach to personal productivity, and the main feature of this essay is the scroll bar down the side. This thing just goes on and on and on, and he talks about how he had a standing desk, but then his heart rate monitor showed him that he gets better health benefits from walking around outside. So he built himself a little like popcorn tray for his laptop. He scanned every document he's ever every document he's ever written from the age of 11. He's got all of this amazing infrastructure setup for being productive, most of which I feel is way over the top for how I want to live my life.

He has a green screen setup in his basement so he can give talks from home.

But there was one thing in here that kind of caught my eye, and that's this idea that he has of a metasearcher. It's a personal search engine that he built that can search every e-mail, every file, every paper he's written, all of the people he knows for any any search term.

And I saw this and I thought, you know, that's something I'd really like. I would love to have a personal search engine for all of my stuff.

But if I'm gonna build that and it's inspired by Stephen Wolfram, I need to come up with a decent name for it.

And so since it is inspired by Wolf Ram, but it's not really aiming to be as good or comprehensive as the thing that he built, I decided to call it Dog Sheep. So I have a project called Dogsheep. I love that pun so much I committed myself to actually building the software.

And so Dogsheep is a collection of tools for personal analytics: tools for pulling in data about yourself from lots of different sources, turning those into SQLite databases so that you can start working with them and then you can load them up in Datasette and start doing interesting things against them.

I'll give you a a demonstration of my personal Dogsheep instance, where all of the data from these tools ends up.

This is my personal Dogsheep. It has data from a a whole bunch of different places. I've got data from Twitter and HealthKit - my Apple Watch tracks my health over time. GitHub data, Foursquare Swarm, I pulled in data from LinkedIn and Goodreads and Hacker News and all of these different places, and I can do all sorts of fun things with this.

So I'll start by showing you a couple of things from from Twitter. So I index all of my tweets, anytime somebody mentions me, and all of the tweets that I've favorited and because I've indexed my favourited tweets I can run searches against them.

So I can search for PyGotham and see all of the tweets that I've favorited about PyGotham.

Twitter does not offer this feature themselves and having this feature suddenly makes Twitter favourites so, so much more useful.

I can search my followers so if I decide I want to have a conversation with a VC, I can type in VC and see anyone who follows me on Twitter who mentions VC in their bio, and hence is somebody that I might be able to contact via direct messages.

But a really useful example is that my dog, Cleo, has a Twitter account. And every time she goes to the vet, she tweets a selfie and with her selfie she tweets how much she weighs. So she'll tweet "I weigh 42.5 pounds. I grew a little bit more dog". Because she tweets this and I've got this coming into Dogsheep I now have these in a SQL database - so I can construct a SQL query that looks for tweets that mentioned her weight. And I can use a regular expression to extract that weight value out as the thing before the LB. And then I can use my charting plugin to plot that as a chart. So I now have a chart of Cleo's weight over time, as reported by her posting selfies at the vet. It's like having the kind of data warehouse you get at a giant corporation, but just against your own data, and costing about $5 a month in hosting fees.

So that's tweets. But there was so much more stuff in here. I'm running dozens and dozens of projects on GitHub, and I pull all of the issues, issue comments, commits and releases into my one database.

So here I have a table of all 7,000 commits that I've made on GitHub across all of these different project so I can search these in one place, I can evaluate them, I can compare commits across different projects and I can also do charting. So here's a graph that's constructed by a custom SQL query, which pulls out just the date of each commit, groups by them and sticks on a count. So these are my commits over time as a chart.

A key philosophy of Datasette is everything should have a URL. So once I've constructed this query and put it on a chart, I can bookmark that page right there and that becomes, essentially, the application encoded in a URL that I can bookmark and link to and visit later on.

I've got a copy of my genome. I did 23AndMe a few years ago, and I found out recently that they have an export button and you can export your genome out as a CSV file. So I did that. I now have a copy of the 600,000 rows of my genome that 23AndMe keep track of. Which means I can use SQL queries to analyse my own genome. I have a query here that tells me what colour my eyes are based on running this query against my genome - apparently my eyes are blue 99% of the time because I have a GG genotype in this particular location on my genome.

This took an entire weekend figure out. I went to Science Hack Day and borrowed a geneticist so I could see if I could figure out some homebrew genetic explorations.

I mentioned the Apple Watch earlier. Apple record an incredible amount of data about me and they don't uploaded to the cloud - an interesting thing about Apple is they keep that stuff on your device. So if you dig around in the HealthKit app on the iPhone, there's an export button that will give you that data back out again. And I wrote code to turn that into SQLite and there was an enormous quantity of data in here. I've got standing time and basal energy burned, my body mass over time reported by my Withing scales, headphone audio exposure is something that showed up last year, the number of flights of steps I climbed, all sorts of stuff

But the most interesting one is workouts because every time you track a workout using your Apple Watch, it goes in the database. And if it's an outdoor workout - where you're, say, going for a run, it records your GPS location every few seconds. I ran the Bay To Breakers race in San Francisco a couple of years ago and beecause of that, I've got a database table with my location tracked route. It's 3,800 points, and I can plot that on a map and see the exact, finely grained route I took through San Francisco when I ran Bay To Brakers just over three years ago.

It's kind of amazing to me that this stuff is just sat there on our devices waiting for us to liberate it and start using it to build these visualisations and learn interesting things about ourselves.

I also mentioned earlier on Apple Photos. Apple Photos databases are absolutely fascinating because it turns out Apple run machine learning models on your phone to figure out what you took photographs of. And this means I can do things like run a SQL query that shows me all of the photographs that I've taken of a pelican.

My personal Dogsheep right here shows me these pictures - these are Apple Photos I took where Apple itself has labelled them as pelican.

And even more fun than that is Apple calculates scores for your photos. These are hidden away in the SQLite database. And they have names like Z Overall Aesthetic Score or Z Harmonious Colour Score or Z Pleasant Camera Tilt Score. So this query here isn't just showing me photographs I've taken of pelicans - it's sorting them by the most aesthetically pleasing according to Apple's classification algorithm that they're running on my device.

I can show you the Pelican photo with the most pleasant camera tilt - there's so much fun stuff you can do digging into all of this.

So I've been building this set of Dogsheep tools for about a year now, but the thing I always want to do was to tie them all together. I wanted to build that personal search engine.

And a couple of weeks ago, I finally got it working.

I've called it Dogsheep Beta because Stephen Wolfram's search engine is called Wolfram Alpha. So I figured Dogsheep Beta was definitely the appropriate name for it.

And so this right here is Dogsheep Beta, and it lets me run searches across my tweets, my photos, my GitHub commits, issue comments, releases, comments I made on Hacker News - all in one place.

And so if I search for example, for "cleopaws costume", I will see all of the tweets by my dog, whose Twitter handle is @Cleopaws where she's tweeting about her costumes. I think she makes a particularly fetching pirate.

Or again, I can search for PyGotham and I get back not just not just tweets, but I get back issue comments from my personal issues where I've been tracking the fact that I need to submit a talk.

Again, this is a custom search interface, but really all this is under the hood is SQLite - it's a SQLite database with a table that acts as a search index and I'm running queries using SQLite's built-in full text search mechanism.

If any of this has caught your interest, all of this is aggressively open source. So if you go to dogsheep.github.io this is an index page for all of the Dogsheep project. Click through to the GitHub Datasette repository and I've got very extensive documentation covering Datasette, what it can do, the plugins available and how to write plugins of your own.

And really, my goal with this project is I want to build a growing ecosystem of not just Datasette plugins, but tools for getting data from all of these different sources and loading them into SQLite databases so that we can join them together and use them to find out interesting things about the world and about our own lives.

If this has piqued your interest, please drop by the project. We have a GitHub discussions board which you're welcome to join.

[I hadn't launched the full website when I gave this talk, but today I'd encourage you to explore datasette.io]


Ben Werdmüller

10 assertions about the future of social

It will be decentralized. The way to compete with Facebook is not to compete with its model. Decentralization empowers users in a way that Facebook can’t, while also undermining its core business model. It won’t be a decentralized version of something we already know. It’s not enough to build a “decentralized Facebook / Twitter / TikTok / whatever”. While there’s a core that’s attracted

It will be decentralized. The way to compete with Facebook is not to compete with its model. Decentralization empowers users in a way that Facebook can’t, while also undermining its core business model.

It won’t be a decentralized version of something we already know. It’s not enough to build a “decentralized Facebook / Twitter / TikTok / whatever”. While there’s a core that’s attracted to the ethos of decentralization, it can’t be the core value prop for most users. It’s got to bring something new in itself.

It won’t be a monoculture. Lots of different clients written by lots of different teams with lots of different user experiences.

It doesn’t have to be web-based. As much as we love the web, we shouldn’t be constrained by having to build web-first. Apps like TikTok - and before it, Instagram - have shown that you can take advantage of native app platforms first, and use web for discovery second.

We can’t solve identity. There will never be a single identity that we use across the web. Instead, there may be open protocols that allow us to auth with different providers.

It won’t be built by an existing tech company. (Although it’s possible it could be built by a spinoff or protected internal team of one.) And it might not be built by a company at all: it’s likely to start as an open source collaboration, or even as a co-operative.

It won’t be built by an existing standards body. And any attempt to build standards-first will fail.

The ecosystem will be simpler than you imagine. The technologies that succeed will allow new developers to get up and running in an afternoon, armed with great documentation, easy-to-use libraries, and simple underlying protocols.

It won’t be based on blockchain. But there’s nothing to say that you won’t be able to bring your own ENS domain, etc, if you want.

It has to solve harassment and abuse. Any new network that doesn’t solve harassment and abuse will be co-opted by right wing groups and trolls, killing any nascent community before it has had a chance to get off the ground.


John Philpin : Lifestream

MIT Predicted in 1972 That Society Will Collapse This Centur

MIT Predicted in 1972 That Society Will Collapse This Century. New Research Shows We’re on Schedule.

@_Nat Zone

東京オリンピックのショーディレクター選択に見られる組織委員会のガバナンスの欠如

(この記事は22日11時03分公開のFaceboo… The post 東京オリンピックのショーディレクター選択に見られる組織委員会のガバナンスの欠如 first appeared on @_Nat Zone.

(この記事は22日11時03分公開のFacebook記事からの転載です)
五輪のショーディレクター小林賢太郎氏のホロコーストギャグについて、サイモン・ウィーゼンタール・センターが動きましたね。

ヤフーニュースの今井佐緒里さんの記事では「緊急請願:菅首相、今日中に五輪のショーディレクター小林氏の処分を。東京五輪が永遠の汚名となる前に」としていますが1、その人がプロデュースしたショーをオリパラでやっても後から刺されかねないし、もはやショーは中止、黙祷で替える2とかじゃないですかね。コロナ犠牲者、震災犠牲者、過去のそして今も続くホロコースト3の犠牲者に向けた。

あと「処分」ですが、前から言っておりますが、これら一連のことは、オリンピック組織委員会のガバナンスの問題であることを如実に表しています4。なので、現在のガバナンス体制に賛成していた人たちはトップ含めてがっさり替えないと駄目なんだと思います。

以下、時系列を追記して行きます 2021-07-22 1:11 中山防衛副大臣への相談からSWCへの照会等に至る一連の経緯のツイート 4:12 高橋浩祐ユダヤ人大量惨殺ごっこ」五輪開会式ディレクターの小林賢太郎氏、芸人時代にホロコーストを笑いのネタに」 8:44 今井佐緒里 「緊急請願:菅首相、今日中に五輪のショーディレクター小林氏の処分を。東京五輪が永遠の汚名となる前に」 10:50 @_nat 「SWCが動いた。だからさ、オリンピック組織委員会のガバナンスの問題なんだって。上からバッサリ変えないと駄目。」 11:03 本記事 12:40 ロイター「東京五輪、開会式前日に演出担当を解任 内容を見直し」 12:48 虚構新聞「【速報】開会式、閉会式後に開催へ。トラブル続きで差し替え間に合わず(虚構新聞社/12:48発表)」 13:03 共同通信「開会式はやらせていただきたいと橋本会長

The post 東京オリンピックのショーディレクター選択に見られる組織委員会のガバナンスの欠如 first appeared on @_Nat Zone.

Wednesday, 21. July 2021

Jon Udell

A virtuous cycle for analytics

Suppose you’re a member of a team that runs a public web service. You need to help both internal and external users make sense of all the data that’s recorded as it runs. That’s been my role for the past few years, now it’s time to summarize what I’ve learned. The web service featured in … Continue reading A virtuous cycle for analytics

Suppose you’re a member of a team that runs a public web service. You need to help both internal and external users make sense of all the data that’s recorded as it runs. That’s been my role for the past few years, now it’s time to summarize what I’ve learned.

The web service featured in this case study is the Hypothesis web annotation system. The primary database, Postgres, stores information about users, groups, documents, courses, and annotations. Questions that our team needs to answer include:

– How many students created annotations last semester?

– In how many courses at each school?

Questions from instructors using Hypothesis in their courses include:

– Which passages in course readings are attracting highlights and discussion?

– Who is asking questions about those passages, and who is responding?

Early on we adopted a tool called Metabase that continues to be a pillar of our analytics system. When Metabase was hooked up to our Postgres database the team could start asking questions without leaning on developers. Some folks used the interactive query builder, while others went straight to writing SQL that Metabase passes through to Postgres.

Before long we had a large catalog of Metabase questions that query Postgres and display results as tables or charts that can be usefully arranged on Metabase dashboards. It’s all nicely RESTful. Interactive elements that can parameterize queries, like search boxes and date pickers, map to URLs. Queries can emit URLs in order to compose themselves with other queries. I came to see this system as a kind of lightweight application server in which to incubate an analytics capability that could later be expressed more richly.

Over time, and with growing amounts of data, early success with this approach gave way to two kinds of frustration: queries began to choke, and the catalog of Metabase questions became unmanageable. And so, in the time-honored tradition, we set up a data warehouse for analytics. Ours is another instance of Postgres that syncs nightly with the primary database. There are lots of other ways to skin the cat but it made sense to leverage ops experience with Postgres and I had a hunch that it would do well in this role.

To unthrottle the choking queries I began building materialized views that cache the results of Postgres queries. Suppose a query makes use of available indexes but still takes a few minutes, or maybe even an hour, to run. It still takes that long to build the corresponding materialized view, but once built other queries can use its results immediately. Metabase questions that formerly included chunks of SQL began reducing to select * from {viewname}.

This process continues to unfold in a highly productive way. Team members may or may not hit a performance wall as they try to use Metabase to answer their questions. When they do, we can convert the SQL text of a Metabase question to a Postgres materialized view that gets immediate results. Such views can join with others, and/or with underlying tables, in SQL SELECT contexts. The views become nouns in a language that expresses higher-order business concepts.

The verbs in this language turned out to be Postgres functions written in the native procedural language, pl/pgsql, and later also in its Python counterpart, pl/python. Either flavor can augment built-in Postgres library functions with user-defined functions that can return simple values, like numbers and strings, but can also return sets that behave in SQL SELECT contexts just like tables and views.

Functions were, at first, a way to reuse chunks of SQL that otherwise had to be duplicated across Metabase questions and Postgres CREATE MATERIALIZED VIEW statements. That made it possible to streamline and refactor both bodies of code and sanely maintain them.

To visualize what had now become a three-body system of sources in which Metabase questions, Postgres views, and Postgres functions can call (or link to) one another, I wrote a tool that builds a crosslinked concordance. That made it practical to reason effectively about the combined system.

Along the way I have learned how Postgres, and more broadly modern SQL, in conjunction with a tool like Metabase, can enable a team like ours to make sense of data. There’s plenty to say about the techniques I’ve evolved, and I aim to write them up over time. The details won’t interest most people, but here’s an outcome that might be noteworthy.

Team member: I had an idea that will help manage our communication with customers, and I’ve prototyped it in a Metabase question.

Toolsmith: Great! Here’s a Postgres function that encapsulates and refines your SQL. It’s fast enough for now, but if needed we can convert it into a materialized view. Now you can use that function in another Metabase question that projects your SQL across a set of customers that you can select.

That interaction forms the basis of a virtuous cycle: The team member formulates a question and does their best to answer it using Metabase; the toolsmith captures the intent and re-expresses it in a higher-level business language; the expanded language enables the team member to go farther in a next cycle.

We recognize this software pattern in the way application programmers who push a system to its limits induce systems programmers to respond with APIs that expand those limits. I suppose it’s harder to see when the application environment is Metabase and the systems environment is Postgres. But it’s the same pattern, and it is powerful.


1 https://blog.jonudell.net/2021/07/21/a-virtuous-cycle-for-analytics/
2 https://blog.jonudell.net/2021/07/24/pl-pgsql-versus-pl-python-heres-why-im-using-both-to-write-postgres-functions/
3 https://blog.jonudell.net/2021/07/27/working-with-postgres-types/


Simon Willison

Weeknotes: sqlite-transform 1.1, Datasette 0.58.1, datasette-graphql 1.5

Work on Project Pelican inspires new features and improvements across a number of different projects. Project Pelican is an upcoming collaborative data release I'm working on with an embargo - hence the code name. It involves around 700MB of CSV data converted to SQLite, which has lead to some small but notable improvements in several of my tools. Datasette 0.58 and 0.58.1 I published annota

Work on Project Pelican inspires new features and improvements across a number of different projects.

Project Pelican is an upcoming collaborative data release I'm working on with an embargo - hence the code name. It involves around 700MB of CSV data converted to SQLite, which has lead to some small but notable improvements in several of my tools.

Datasette 0.58 and 0.58.1

I published annotated release notes for Datasette 0.58 last week. In addition to several plugin hooks improvement the big improvement was a 10x performance improvement to Datasette facets, which was discovered during work on Project Pelican. Conversations about this on the SQLite Forum lead to a new optimization by D. Richard Hipp landing in SQLite trunk!

I release Datasette 0.58.1 shortly afterwards with a fix for a rare intermittent race condition that I introduced back in January.

sqlite-transform 1.0 and 1.1

The data I am working with for Project Pelican has a confusing characteristic: some of the columns contain integers almost consistently, but have a few rows that contain text: a - character, or a number with commas like 1,232 or a string such as >40).

I decided to address this by adding more columns - the original column with a _int suffix of type integer containing just the number, if one is available. So if the column was called total there would be a total_int integer column reflecting its value.

My sqlite-transform tool could almost do this: it has a lambda command that lets you execute Python code against every value in a column to convert it in some way.

In sqlite-transform 1.0 I added the ability to write that converted value to a different output column - so now I can do this:

sqlite-transform lambda project-pelican.db \ sometable total \ --code ' if ( value.isdigit() or value.replace(",", "").isdigit() or value.startswith("<"): return int(value.replace(",", "").replace("<", "")) ' --output total_int --output-type integer

This command-line script compiles the Python code in --code once and then executes it against every value in the total column. The return value is written to the newly created total_int column.

Under the hood this works by defining a custom SQLite function called transform() using the compiled Python code, then executing the SQL equivalent of the following:

update sometable set total_int = transform(total)

The Python function tracks how many times it has been called and uses that to update a progress bar using the tqdm Python library.

sqlite-transform 1.1 added a --silent option to disable that progress bar, because when I ran it in GitHub Actions the progress bar output turned into too many separate lines of text.

conditional-get 0.3

The Project Pelican Datasette instance is built by GitHub Actions, which starts by downloading a 700MB CSV file.

I decided to use the GitHub Actions cache in conjunction with a conditional HTTP GET request to avoid downloading the CSV file if it hadn't changed since the last time the action ran.

My conditional-get tool can help here: it downloads files but stores their ETag in a JSON file, then on subsequent requests it uses that ETag to perform a conditional GET and avoids downloading the content of the target URL if it has not changed since last time.

One catch though: the CSV files for Project Pelican live in Google Cloud Storage and are protected by a signed URL. The JSON file used the full URL as the key to lookup an ETag, so if the URL has changed (due to a fresh signature) the tool would not be able to find the correct ETag.

conditional-get 0.3 fixes that with a new --key option that can be used to specify a custom key within the JSON file to store the ETag. So now my GitHub Action script partially looks like this:

name: Build and deploy database on: push: branches: - main workflow_dispatch: jobs: build_and_deploy: runs-on: ubuntu-latest steps: - name: Check out repo uses: actions/checkout@v2 - name: Set up Python uses: actions/setup-python@v2 with: python-version: 3.9 - uses: actions/cache@v2 name: Configure pip caching with: path: ~/.cache/pip key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} restore-keys: | ${{ runner.os }}-pip- - uses: actions/cache@v2 name: Configure HTTP download caching with: path: ~/data key: ${{ runner.os }}-http-download-cache - name: Install Python dependencies run: | pip install -r requirements.txt - name: Download CSVs env: JWT: ${{ secrets.JWT }} run: | DOWNLOAD_DIR=~/data ./download.sh

Where requirements.txt contains conditional-get and download.sh looks like this:

#!/bin/bash set -euf -o pipefail download_dir="${DOWNLOAD_DIR:-.}" mkdir -p $download_dir filename=states.csv signed_url=$(curl -s 'https://api.example.com/graphql' \ -H "authorization: JWT $JWT" \ -H 'content-type: application/json' \ --data-raw $'{"operationName":"CreateFileDownloadURI","variables":{"input":{"fileName":"'$filename$'","projectId":"123"}},"query":"mutation CreateFileDownloadURI($input: FileURIInput\u0021) {\\n createFileDownloadUri(input: $input) {\\n ok {\\n name\\n uri\\n __typename\\n }\\n err\\n __typename\\n }\\n}\\n"}' \ --compressed | jq -r .data.createFileDownloadUri.ok.uri) conditional-get --etags $download_dir/etags.json \ --key $filename -v $signed_url -o $download_dir/$filename

I'm making a GraphQL call here from curl to retrieve the signed URL, then passing that signed URL to the following to perform the conditional GET request:

conditional-get --etags $download_dir/etags.json \ --key $filename -v $signed_url -o $download_dir/$filename datasette-graphql 1.5

Unrelated to Project Pelican (despite that project making calls to a separate GraphQL API): datasette-graphql 1.5 finally closed an issue that had been open for a while: switching internal API calls over to using datasette.client.

Datasette exposes two key documented APIs: there's the HTTP/JSON API that can be used to query and introspect Datasette over the network, and there's the internal Python API which allows plugins to interact with the rest of Datasette.

Last year I realized that there are often times when a plugin might wish to access that same JSON API that is exposed by Datasette to the public.

Datasette uses ASGI internally, and the HTTPX Python library has a mechanism for executing requests directly against a Python ASGI interface without using the network... so I combined the two and introduced the datasette.client interface for making calls from within Datasette to its own HTTP API.

My dogsheep-beta search plugin was the first to take advantage of this: it uses that mechanism to make internal API calls to calculate facet counts without re-implementing faceting.

datasette-graphql exposes a GraphQL API over Datasette tables, including the ability to apply filters (rows where age > 25 etc). It does this by translating GraphQL field requests into calls to the Datasette table JSON API.

This used to work with some inelegant hijacking of Datasette's undocumented internal view classes, which made me nervous because it meant that the GraphQL plugin would break any time I modified Datasette core.

In 1.5 I switched it over to using the datasette.client mechanism instead, which should be a much more stable target since changes to Datasette's JSON API are made in a much more controlled fashion.

In writing the tests for this I ran into the race condition mentioned above - so releasing this plugin ended up being the trigger for pushing out the Datasette 0.58.1 release with that fix.

Releases this week sqlite-transform: 1.1 - (8 releases total) - 2021-07-21
Tool for running transformations on columns in a SQLite database datasette-export-notebook: 1.0 - (5 releases total) - 2021-07-17
Datasette plugin providing instructions for exporting data to Jupyter or Observable datasette-graphql: 1.5 - (31 releases total) - 2021-07-16
Datasette plugin providing an automatic GraphQL API for your SQLite databases datasette: 0.58.1 - (94 releases total) - 2021-07-16
An open source multi-tool for exploring and publishing data conditional-get: 0.3 - (4 releases total) - 2021-07-14
CLI tool for fetching data using HTTP conditional get TIL this week Importing CSV data into SQLite with .import Using the tesseract CLI tool

Ben Werdmüller

Are They Picking At Us?

“Racism and xenophobia can hide under the very thin veneer of polite societal norms, like that shiny, sugary layer on a fruit tart that looks super-fake at the supermarket bakery table. I felt grateful for the refuge the community potluck table at our mosque offered. Ours was the only table in town where you could find grilled seekh kebabs, barbecue chicken, KFC hot wings, Pal

“Racism and xenophobia can hide under the very thin veneer of polite societal norms, like that shiny, sugary layer on a fruit tart that looks super-fake at the supermarket bakery table. I felt grateful for the refuge the community potluck table at our mosque offered. Ours was the only table in town where you could find grilled seekh kebabs, barbecue chicken, KFC hot wings, Palestinian chicken with vermicelli rice, potato salad, okra sabzi, Southern fried okra, Lahori-style fried fish, cornmeal-fried trout, fruit chaat, and, yes, even that fruit tart.”

[Link]


Apple reportedly postpones in-person work until at least October

Likely the first of many. [Link]

Likely the first of many.

[Link]


In-Vehicle Infotainment Systems Especially Distracting to Older Drivers

On car touchscreens and voice interfaces: “Researchers found that the technology created potentially unsafe distractions for all drivers, though this safety risk is more pronounced for older adults, who took longer (4.7-8.6 seconds) to complete tasks, experienced slower response times, and increased visual distractions.” [Link]

On car touchscreens and voice interfaces: “Researchers found that the technology created potentially unsafe distractions for all drivers, though this safety risk is more pronounced for older adults, who took longer (4.7-8.6 seconds) to complete tasks, experienced slower response times, and increased visual distractions.”

[Link]

Tuesday, 20. July 2021

@_Nat Zone

あり得たもう一つのオリンピック開会式

今日、故あって、銀座のエルメス・ギャラリーでやって… The post あり得たもう一つのオリンピック開会式 first appeared on @_Nat Zone.

今日、故あって、銀座のエルメス・ギャラリーでやっているマチュウ・コプランによる展覧会「エキシビジョン・カッティングス」1に行ってきました。アートシーンには全く疎いのですが、前回の東京オリンピックの1964年というと、日本は世界をリードしてたころでなんですね。

大パノラマ展とか2

普通は画廊での個展は初日は華やかにレセプションとかあるものですが、これは画廊を封鎖して始まり、個展の終わりに封鎖を解いて祝って終わるという反展覧会 Anti-Exhabition。それに続く世界各地でのアートの商業化に対するアンチテーゼの動きを見て、色々考えるところがありました。

それの一つが、あり得たもう一つのオリンピック開会式。

コロナでロックダウンの時期だったんだから、それにふさわしく反開会式 Anti-Opening にして、選手は誰も入場せず、オリンピック村の自室のバルコニーにソーシャルディスタンスで並び、国立競技場にはバッハ会長他オリンピック貴族と関係者だけがいるのをずっと画面半分でTV中継、もう半分ではドローンで宿舎バルコニーを国別に写して回る。音楽はケージの4分33秒でコロナでなくなった方々と、「復興五輪」なんだから10年前の震災で亡くなった方々に黙祷を捧げる。後者に関しては、復興の現状を包み隠さず写す。

オリンピックの最中は、IOC貴族たちにカメラが密着。東京オリンピックとしてのアートタッチのドキュメンタリーを残す。

選手たちが一同に会すのはもちろん閉会式の最後までお預けです。残った人たちだけですが。残らなかった人たちは大スクリーンでバーチャル参加ですね。

最後の最後に誰か象徴的な人(誰が良いですかね)がオリンピック貴族たちを排除してアスリートたちが集まり、商業主義との決別を宣言3するとか。LAオリンピック以来ずっとではありますが、今回は特にこれだけ商業主義とアテンションエコノミーの暗黒を見せつけている4オリンピックなのでちょっと妄想してみました。

The post あり得たもう一つのオリンピック開会式 first appeared on @_Nat Zone.

Damien Bod

Using an ASP.NET Core IHostedService to run Azure Service Bus subscriptions and consumers

This post shows how Azure Service bus subscription for topics or consumers for a queue, or can be used inside an ASP.NET Core application. The Azure Service Bus client listens to events and needs to be started, stopped and registered to the topic to receive messages. An IHostedService is used for this. Code: https://github.com/damienbod/AspNetCoreServiceBus Posts […]

This post shows how Azure Service bus subscription for topics or consumers for a queue, or can be used inside an ASP.NET Core application. The Azure Service Bus client listens to events and needs to be started, stopped and registered to the topic to receive messages. An IHostedService is used for this.

Code: https://github.com/damienbod/AspNetCoreServiceBus

Posts in this series:

Using Azure Service Bus Queues with ASP.NET Core Services Using Azure Service Bus Topics in ASP.NET Core Using Azure Service Bus Topics Subscription Filters in ASP.NET Core Using Entity Framework Core to process Azure Service Messages in ASP.NET Core Using an Azure Service Bus Topic Subscription in an Azure Function Using Azure Service Bus with restricted access Using an ASP.NET Core IHostedService to run Azure Service Bus subscriptions and consumers

The ServiceBusTopicSubscription class is used to setup the Azure Service bus subscription. The class uses the ServiceBusClient to set up the message handler, the ServiceBusAdministrationClient is used to implement filters and add or remove these rules. The Azure.Messaging.ServiceBus Nuget package is used to connect to the subscription.

using Azure.Messaging.ServiceBus; using Azure.Messaging.ServiceBus.Administration; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Logging; using System; using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; namespace ServiceBusMessaging { public class ServiceBusTopicSubscription : IServiceBusTopicSubscription { private readonly IProcessData _processData; private readonly IConfiguration _configuration; private const string TOPIC_PATH = "mytopic"; private const string SUBSCRIPTION_NAME = "mytopicsubscription"; private readonly ILogger _logger; private readonly ServiceBusClient _client; private readonly ServiceBusAdministrationClient _adminClient; private ServiceBusProcessor _processor; public ServiceBusTopicSubscription(IProcessData processData, IConfiguration configuration, ILogger<ServiceBusTopicSubscription> logger) { _processData = processData; _configuration = configuration; _logger = logger; var connectionString = _configuration.GetConnectionString("ServiceBusConnectionString"); _client = new ServiceBusClient(connectionString); _adminClient = new ServiceBusAdministrationClient(connectionString); } public async Task PrepareFiltersAndHandleMessages() { ServiceBusProcessorOptions _serviceBusProcessorOptions = new ServiceBusProcessorOptions { MaxConcurrentCalls = 1, AutoCompleteMessages = false, }; _processor = _client.CreateProcessor(TOPIC_PATH, SUBSCRIPTION_NAME, _serviceBusProcessorOptions); _processor.ProcessMessageAsync += ProcessMessagesAsync; _processor.ProcessErrorAsync += ProcessErrorAsync; await RemoveDefaultFilters().ConfigureAwait(false); await AddFilters().ConfigureAwait(false); await _processor.StartProcessingAsync().ConfigureAwait(false); } private async Task RemoveDefaultFilters() { try { var rules = _adminClient.GetRulesAsync(TOPIC_PATH, SUBSCRIPTION_NAME); var ruleProperties = new List<RuleProperties>(); await foreach (var rule in rules) { ruleProperties.Add(rule); } foreach (var rule in ruleProperties) { if (rule.Name == "GoalsGreaterThanSeven") { await _adminClient.DeleteRuleAsync(TOPIC_PATH, SUBSCRIPTION_NAME, "GoalsGreaterThanSeven") .ConfigureAwait(false); } } } catch (Exception ex) { _logger.LogWarning(ex.ToString()); } } private async Task AddFilters() { try { var rules = _adminClient.GetRulesAsync(TOPIC_PATH, SUBSCRIPTION_NAME) .ConfigureAwait(false); var ruleProperties = new List<RuleProperties>(); await foreach (var rule in rules) { ruleProperties.Add(rule); } if (!ruleProperties.Any(r => r.Name == "GoalsGreaterThanSeven")) { CreateRuleOptions createRuleOptions = new CreateRuleOptions { Name = "GoalsGreaterThanSeven", Filter = new SqlRuleFilter("goals > 7") }; await _adminClient.CreateRuleAsync(TOPIC_PATH, SUBSCRIPTION_NAME, createRuleOptions) .ConfigureAwait(false); } } catch (Exception ex) { _logger.LogWarning(ex.ToString()); } } private async Task ProcessMessagesAsync(ProcessMessageEventArgs args) { var myPayload = args.Message.Body.ToObjectFromJson<MyPayload>(); await _processData.Process(myPayload).ConfigureAwait(false); await args.CompleteMessageAsync(args.Message).ConfigureAwait(false); } private Task ProcessErrorAsync(ProcessErrorEventArgs arg) { _logger.LogError(arg.Exception, "Message handler encountered an exception"); _logger.LogDebug($"- ErrorSource: {arg.ErrorSource}"); _logger.LogDebug($"- Entity Path: {arg.EntityPath}"); _logger.LogDebug($"- FullyQualifiedNamespace: {arg.FullyQualifiedNamespace}"); return Task.CompletedTask; } public async ValueTask DisposeAsync() { if (_processor != null) { await _processor.DisposeAsync().ConfigureAwait(false); } if (_client != null) { await _client.DisposeAsync().ConfigureAwait(false); } } public async Task CloseSubscriptionAsync() { await _processor.CloseAsync().ConfigureAwait(false); } } }

The WorkerServiceBus class implements the IHostedService interface and uses the IServiceBusTopicSubscription interface to subscribe to an Azure Service Bus topic. The StartAsync method is used to register the subscription using the RegisterOnMessageHandlerAndReceiveMessages method. The interface provides a start, and stop and a dispose. The Azure Service Bus class is controlled using this hosted service. If needed, a periodic task could be implemented to run health checks on the client or whatever.

public class WorkerServiceBus : IHostedService, IDisposable { private readonly ILogger<WorkerServiceBus> _logger; private readonly IServiceBusConsumer _serviceBusConsumer; private readonly IServiceBusTopicSubscription _serviceBusTopicSubscription; public WorkerServiceBus(IServiceBusConsumer serviceBusConsumer, IServiceBusTopicSubscription serviceBusTopicSubscription, ILogger<WorkerServiceBus> logger) { _serviceBusConsumer = serviceBusConsumer; _serviceBusTopicSubscription = serviceBusTopicSubscription; _logger = logger; } public async Task StartAsync(CancellationToken stoppingToken) { _logger.LogDebug("Starting the service bus queue consumer and the subscription"); await _serviceBusConsumer.RegisterOnMessageHandlerAndReceiveMessages().ConfigureAwait(false); await _serviceBusTopicSubscription.PrepareFiltersAndHandleMessages().ConfigureAwait(false); } public async Task StopAsync(CancellationToken stoppingToken) { _logger.LogDebug("Stopping the service bus queue consumer and the subscription"); await _serviceBusConsumer.CloseQueueAsync().ConfigureAwait(false); await _serviceBusTopicSubscription.CloseSubscriptionAsync().ConfigureAwait(false); } public void Dispose() { Dispose(true); GC.SuppressFinalize(this); } protected virtual async void Dispose(bool disposing) { if (disposing) { await _serviceBusConsumer.DisposeAsync().ConfigureAwait(false); await _serviceBusTopicSubscription.DisposeAsync().ConfigureAwait(false); } } }

The IHostedService is added to the services in the ConfigureServices method. The AddHostedService is used to initialize this. Now the Azure Service bus subscription can be managed and consume messages from the topic subscription or a queue is used.

public void ConfigureServices(IServiceCollection services) { services.AddControllers(); var connection = Configuration.GetConnectionString("DefaultConnection"); services.AddDbContext<PayloadContext>(options => options.UseSqlite(connection)); services.AddSingleton<IServiceBusConsumer, ServiceBusConsumer>(); services.AddSingleton<IServiceBusTopicSubscription, ServiceBusTopicSubscription>(); services.AddSingleton<IProcessData, ProcessData>(); services.AddHostedService<WorkerServiceBus>(); services.AddSwaggerGen(c => { c.SwaggerDoc("v1", new OpenApiInfo { Version = "v1", Title = "Payload API", }); }); }

When the application is run, the messages can be sent to the topic and are received using the IHostedService Azure Service Bus subscription.

Links:

https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/servicebus/Azure.Messaging.ServiceBus

https://docs.microsoft.com/en-us/azure/service-bus-messaging/

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services

Monday, 19. July 2021

Simon Willison

Launch HN Instructions

Launch HN Instructions The instructions for YC companies that are posting their launch announcement on Hacker News are really interesting to read. "As founders, you're used to talking to users, customers, and investors. HN readers are not any of those—what they are is peers, and using any of those styles with peers feels clueless and entitled. [...] To interest HN, write in a factual, personal,

Launch HN Instructions

The instructions for YC companies that are posting their launch announcement on Hacker News are really interesting to read. "As founders, you're used to talking to users, customers, and investors. HN readers are not any of those—what they are is peers, and using any of those styles with peers feels clueless and entitled. [...] To interest HN, write in a factual, personal, and modest way about what problem you solve, why it matters, how you solve it, and how you got there."

Via dang


toyDB: references

toyDB: references toyDB is a "distributed SQL database in Rust, written as a learning project", with its own implementations of SQL, raft, ACID transactions, B+trees and more. toyDB author Erik Grinaker has assembled a detailed set of references that he used to learn how to build a database - I'd love to see more projects do this, it's really useful.

toyDB: references

toyDB is a "distributed SQL database in Rust, written as a learning project", with its own implementations of SQL, raft, ACID transactions, B+trees and more. toyDB author Erik Grinaker has assembled a detailed set of references that he used to learn how to build a database - I'd love to see more projects do this, it's really useful.


Inserting One Billion Rows in SQLite Under a Minute

Inserting One Billion Rows in SQLite Under a Minute Avinash Sajjanshetty experiments with accelerating writes to a test table in SQLite, using various SQLite pragmas to accelerate inserts followed by a rewrite of Python code to Rust. Also of note: running the exact same code in PyPy saw a 3.5x speed-up! Via Hacker News

Inserting One Billion Rows in SQLite Under a Minute

Avinash Sajjanshetty experiments with accelerating writes to a test table in SQLite, using various SQLite pragmas to accelerate inserts followed by a rewrite of Python code to Rust. Also of note: running the exact same code in PyPy saw a 3.5x speed-up!

Via Hacker News

Sunday, 18. July 2021

Simon Willison

Organize and Index Your Screenshots (OCR) on macOS

Organize and Index Your Screenshots (OCR) on macOS Alexandru Nedelcu has a very neat recipe for creating an archive of searchable screenshots on macOS: set the default save location for screenshots to a Dropbox folder, then create a launch agent that runs a script against new files in that folder to run tesseract OCR to convert them into a searchable PDF. Via @alexelcu

Organize and Index Your Screenshots (OCR) on macOS

Alexandru Nedelcu has a very neat recipe for creating an archive of searchable screenshots on macOS: set the default save location for screenshots to a Dropbox folder, then create a launch agent that runs a script against new files in that folder to run tesseract OCR to convert them into a searchable PDF.

Via @alexelcu


Quoting David Beazley

I've always believed that a book, even a technical book, should try to tell a cohesive story. The challenge is that as Python has grown in popularity, it has really turned into three different languages--each with their own story. There is a whimsical Python for scripting and tinkering, a quirky Python for asynchronous programming and networking, and a serious Python for enterprise applications.

I've always believed that a book, even a technical book, should try to tell a cohesive story. The challenge is that as Python has grown in popularity, it has really turned into three different languages--each with their own story. There is a whimsical Python for scripting and tinkering, a quirky Python for asynchronous programming and networking, and a serious Python for enterprise applications. Sometimes these stories intersect. Sometimes not.

David Beazley

Saturday, 17. July 2021

Simon Willison

It doesn't take much public creativity to stand out as a job candidate

I've spent nearly twenty years blogging, giving talks and releasing open source code. It's been fantastic for my career, and a huge amount of work. But here's a useful secret: you don't have to put very much work at all into public creativity in order to stand out as a job candidate. I've interviewed hundreds of people, and screened hundreds more resumes - mostly for tech roles in San Francisco,

I've spent nearly twenty years blogging, giving talks and releasing open source code. It's been fantastic for my career, and a huge amount of work. But here's a useful secret: you don't have to put very much work at all into public creativity in order to stand out as a job candidate.

I've interviewed hundreds of people, and screened hundreds more resumes - mostly for tech roles in San Francisco, an extremely competitive job market.

The vast majority of candidates have little to no evidence of creativity in public at all. The same is true for many of the best engineers I have worked with.

As a hiring manager, this means you have to learn how to source candidates and interview effectively: you don't want to miss out on a great engineer just because they spent all of their energy making great products for prior employers rather than blogging, speaking and coding in public.

But as a candidate, this means you can give yourself a big advantage in terms of standing out from the crowd with a relatively small amount of work.

Start a blog. Post an interesting technical article to it once or twice a year - something you've learned, or a bug you've fixed, or a problem you've solved. After a few years stop bothering entirely, but leave the blog online somewhere.

Build a small personal project and put the code on GitHub. Accompany it with a README with a detailed description of the project and screenshots of it in action - almost no-one does this, it only takes a few hours extra and it massively increases the impact your project will have on hiring managers who are checking you out.

That's it. One or two blog posts. Maybe a GitHub repository. Believe it or not, if you are up against a bunch of other candidates (especially earlier on in your career) they likely won't have anything like that. You will jump straight to the top of the hiring manager's mental list, maybe without them even noticing.

There's plenty more you can do if you want to put the effort in: build an audience on Twitter, start a newsletter, make videos, give talks (ideally that get recorded and published online), release open source packages, publish TILs - but honestly if your goal is to get through the interview process more easily you will very quickly hit the law of diminishing returns.

If you're going to do that stuff do it because you want to develop those skills and share with and learn from the world - don't just do it because you think it's a critical path to being hired.

This post started out as a Twitter thread:

If you want to stand out from other candidates, having even one piece of writing or published piece of code that shows something you've built is a great way to do that https://t.co/QfYEWxfIet

- Simon Willison (@simonw) July 17, 2021

Datasette downloads per day (with Observable Plot)

Datasette downloads per day (with Observable Plot) I built an Observable notebook that imports PyPI package download data from datasette.io (itself scraped from pypistats.org using a scheduled GitHub Action) and plots it using Observable Plot. Datasette downloads from PyPI apparently jumped from ~800/day in May to ~4,000/day in July - would love to know why! Via @simonw

Datasette downloads per day (with Observable Plot)

I built an Observable notebook that imports PyPI package download data from datasette.io (itself scraped from pypistats.org using a scheduled GitHub Action) and plots it using Observable Plot. Datasette downloads from PyPI apparently jumped from ~800/day in May to ~4,000/day in July - would love to know why!

Via @simonw


The Digital Antiquarian: Sam and Max Hit the Road

The Digital Antiquarian: Sam and Max Hit the Road Delightful history and retrospective review of 1993's Sam and Max Hit the Road. I didn't know Sam and Max happened because the independent comic's creator worked for LucasArts and the duo had embedded themselves in LucasArts culture through their use in the internal educational materials prepared for SCUMM University.

The Digital Antiquarian: Sam and Max Hit the Road

Delightful history and retrospective review of 1993's Sam and Max Hit the Road. I didn't know Sam and Max happened because the independent comic's creator worked for LucasArts and the duo had embedded themselves in LucasArts culture through their use in the internal educational materials prepared for SCUMM University.


Last Mile Redis

Last Mile Redis Fly.io article about running a local redis cache in each of their geographic regions - "Cache data overlaps a lot less than you assume it will. For the most part, people in Singapore will rely on a different subset of data than people in Amsterdam or São Paulo or New Jersey." But then they note that Redis has the ability to act as both a replica of a primary AND a writable server

Last Mile Redis

Fly.io article about running a local redis cache in each of their geographic regions - "Cache data overlaps a lot less than you assume it will. For the most part, people in Singapore will rely on a different subset of data than people in Amsterdam or São Paulo or New Jersey." But then they note that Redis has the ability to act as both a replica of a primary AND a writable server at the same time ("replica-read-only no"), which actually makes sense for a cache - it lets you cache local data but send out cluster-wide cache purges if necessary.

Via Hacker News

Friday, 16. July 2021

Kerri Lemole

The Future of Open Badges is Verifiable

‘hexadaisy layout’ by Darla available at http://www.flickr.com/photos/pedrosz/2040577615 under a Attribution 2.0 Generic (CC BY 2.0) license When Open Badges was kicking off ten years ago (see the original white paper), it was conceived to be a recognition infrastructure for skills attained and achievements accomplished anywhere at any time. Badges could assert skills learned informally, forma
‘hexadaisy layout’ by Darla available at http://www.flickr.com/photos/pedrosz/2040577615 under a Attribution 2.0 Generic (CC BY 2.0) license

When Open Badges was kicking off ten years ago (see the original white paper), it was conceived to be a recognition infrastructure for skills attained and achievements accomplished anywhere at any time. Badges could assert skills learned informally, formally, really in any aspect of and through life. It was hoped that recruiters, employers, and others could evaluate the badges to find people who had skills that aligned with the opportunities being offered. It was envisioned they could work like other types of credentials such as degrees and certifications, “but with room for much more granular or diverse skill representation” and available opportunities to capture these skills.

The infrastructure for this type of recognition system was a format called a digital badge. More than just a digital sticker, the badges would be filled with metadata properties describing the skills and achievements. Then the badges could convey the metadata both as human readable and machine readable content. By having a standard to describe the metadata and how it would be verified, Open Badges became usable in many technologies and contexts. For the most part, this resulted in badges being shared online on issuing platforms and social media — more as a tool for human understanding than one that took full advantage of the potential for machine readability.

Since then, the ethos of Open Badges, recognizing skills anywhere and anytime, has strengthened. Growth has accelerated, especially in the past few years. IMS Global and Credential Engine’s Badge Count 2020 Report indicated that there’d been an 80% increase in issued badges since 2018. Its use has expanded into more formal institutional contexts, the metadata have evolved, and initiatives have risen around it including the Open Recognition Alliance, Badge Summit, and the Open Skills Network.

Over the years, internet technologies have evolved too and a credential verification model at the W3C called Verifiable Credentials has gained traction. This verification model can be used for any type of credential including passports, drivers licenses, and educational credentials like Open Badges. In fact, members of the Open Badges community helped to write the very first use cases for Verifiable Credentials back in 2018. This was because we knew then that if Open Badges data were to be trusted by apps, they not only needed to be readable by machines, they needed to be verifiable.

The Verifiable Credential model can provide security and privacy enhancements not yet available to Open Badges. This model gives learners persistent access to their badges increasing the longevity and resilience of credentials that were intended to promote and support lifelong learning. A more secure and universally supported verification model such as Verifiable Credentials enables Open Badges to become the personal skills currency it was originally envisioned to be.

Concentric Sky (Badgr) has offered a proposal for a new version of Open Badges that explains what is needed to make it work within the Verifiable Credentials model. The use cases in the proposal provide scenarios where Open Badges are earned and then how they are exchanged. In one use case, a medical professional is able to start her new job sooner because her CME’s can be quickly verified:

Verifying Continuing Ed: Denise was offered a new job at a hospital as a physician assistant. Before starting, her continuing education training and license to practice needed to be verified. The last time she switched hospitals, the verification process took three weeks. This time, she was able to provide her badges to prove her training and license. Within minutes her credentials were verified and she was issued a new digital staff credential

In another use case, a career change is facilitated by using verifiable Open Badges to map skills to jobs:

Mapping Skills: Sid is shifting careers after many years working in construction. In his digital wallet he had several skill badges describing his mastery of several skills in construction but also in teamwork, communication, and organizational skills. Sid also had badges from some courses he’d taken in science and math over the last few years. After he uploaded the skill and course badges from his wallet to a career planning site, he was offered several opportunities to apply for work in software sales and cybersecurity.

Here’s an example of how an Open Badge as a Verifiable Credential (Open Badges 3.0) exchange could work:

A learner connects to a badge issuing platform with their digital wallet app on their phone or laptop. Once authenticated, the issuer provides the badge to the learner who puts it in their wallet. The badge data contains cryptographic proof that identifies the issuer and the learner. A job employment app asks for proof that the applicant has experience with a requirement needed for that role. The learner presents the job employment app with the badge using the digital wallet app. The job employment app can then verify
a. that the learner providing the badge is the recipient of that badge
b. that the issuer is the identity that issued the badge, and
c. that the badge data has not changed since it was issued. The verifier responds that the badge is authentic.

In comparison, here’s an Open Badges 2.0 flow:

A learner or an organization provides an issuer app with the learner’s email address The Issuer generates badge data that includes the email address as the recipient identity and sends the earner the badge (typically as a link to a web page) The earner can share the link on social media, or perhaps with a potential employer or a job application app. The badge is verified by either
a. a human looking at the web page where the badge is hosted or
b. the application attempts to retrieve the badge data from a url hosted by the issuer.

The Open Badges 2.0 example depends on the issuer hosting the data and relies on an email address for the learner. The Open Badges 3.0 example is self-contained and doesn’t require the issuer to continue to retain a web hosting provider in order for the credential to remain valid. Instead it uses cryptographic proof to authenticate the badge, the issuer, as well as the learner who earned it. With either example, the learner has a way to proudly share their achievement online but the Open Badges 3.0 method doesn’t rely on that online presence for verification. In fact, the original issuer may no longer exist, but the achievements can still be verified.

On Monday July 19, we’ll be reviewing the Open Badges 3.0 proposal and anyone is invited to join us to learn more. Here’s the meeting info:

Monday, July 19, 2021
Time: 8am PDT / 11am EDT / 4pm BST, 5pm CEST
Jitsi Web Conference: https://meet.w3c-ccg.org/education
US phone: tel:+1.602.932.2243;3

On Thursday, July 22, Concentric Sky will be presenting this proposal to the IMS Global Open Badges working group to seek the members’ go-ahead to move forward on the work to make Open Badges 3.0 a reality. Public comments may be submitted here.


Simon Willison

The Untold Story of SQLite With Richard Hipp

The Untold Story of SQLite With Richard Hipp This is a really interesting interview with SQLite creator D. Richard Hipp - it covers all sorts of aspects of the SQLite story I hadn't heard before, from its inspiration by a software challenge on a battleship to the first income from clients such as AOL and Symbian to the formation of the SQLite Consortium (based on advice from Mozilla's Mitchell B

The Untold Story of SQLite With Richard Hipp

This is a really interesting interview with SQLite creator D. Richard Hipp - it covers all sorts of aspects of the SQLite story I hadn't heard before, from its inspiration by a software challenge on a battleship to the first income from clients such as AOL and Symbian to the formation of the SQLite Consortium (based on advice from Mozilla's Mitchell Baker) and more.


Datasette 0.58: The annotated release notes

I released Datasette 0.58 last night, with new plugin hooks, Unix domain socket support, a major faceting performance fix and a few other improvements. Here are the annotated release notes. Faceting performance improvement Facets remains my favourite feature in Datasette: it turns out a simple group by / count against a column is one of the most productive ways I know of to start understanding

I released Datasette 0.58 last night, with new plugin hooks, Unix domain socket support, a major faceting performance fix and a few other improvements. Here are the annotated release notes.

Faceting performance improvement

Facets remains my favourite feature in Datasette: it turns out a simple group by / count against a column is one of the most productive ways I know of to start understanding new data.

Yesterday I stumbled across a tiny tweak (details in this issue) that gave me a 10x performance boost on facet queries! Short version: given the following example query:

select country_long, count(*) from ( select * from [global-power-plants] order by rowid ) where country_long is not null group by country_long order by count(*) desc

Removing the unnecessary order by rowid from that inner query knocked the time down from 53ms to 7.2ms (and makes even more of a difference on larger tables).

I was surprised SQLite didn't perform that optimization automatically - so I started a thread on the SQLite forum and SQLite author D. Richard Hipp figured out a patch! It's not yet certain that it will land in a SQLite release but I'm excited to have found an issue interesting enough to be worth looking into. (UPDATE: it landed on trunk).

The get_metadata() plugin hook

New plugin hook: get_metadata(datasette, key, database, table), for returning custom metadata for an instance, database or table. Thanks, Brandon Roberts! (#1384)

Brandon Roberts contributed this hook as part of work he's been doing with Newsday nextLI - always exciting to see Datasette used by another news organization. Brandon has a live demo of the plugins he has been building: datasette-live-config, datasette-live-permissions, datasette-csv-importer and datasette-surveys. He also has a 6 minute demo video explaining the project so far.

The new hook allows plugins to customize the metadata displayed for different databases and tables within the Datasette interface.

There is one catch at the moment: the plugin doesn't yet allow for async calls (including await db.execute(sql)) because Datasette's own internals currently treat access to metadata as a sync rather than async feature.

There are workarounds for this. Brandon's datasette-live-config plugin opens an additional, synchronous connection to the DB which is completely fine for fast queries. Another option would be to keep metadata in an in-memory Python dictionary which is updated by SQL queries that run in an async background task.

In the longer run though I'd like to redesign Datasette's internals to support asynchronous metadata access - ideally before Datasette 1.0.

The skip_csrf() plugin hook

New plugin hook: skip_csrf(datasette, scope), for opting out of CSRF protection based on the incoming request. (#1377)

I wanted to write a plugin that supported an HTTP POST to a Datasette form that wasn't protected by Datasette's CSRF protection. This proved surprisingly difficult! I ended up shipping asgi-csrf 0.9 with a new mechanism for custom opting-out of CSRF protection based on the ASGI scope, then exposing that mechanism in a new plugin hook in Datasette.

CSRF is such a frustrating security issue to write code against, because in modern browsers the SameSite cookie attribute more-or-less solves the problem for you... but that attribute only has 90% global usage according to caniuse.com - not quite enough for me to forget about it entirely.

There also remains one obscure edge-case in which SameSite won't help you: the definition of "same site" includes other subdomains of your domain (provided it's not on the Public Suffix List). This means that for SameSite CSRF protection to work you need to be confident that no subdomains of your domain will suffer an XSS - and in my experience its common for subdomains to be pointed at third-party applications that may not have the same stringent XSS protection as your main code.

So I continue to care about CSRF protection in Datasette.

Unix domain socket support

New datasette --uds /tmp/datasette.sock option for binding Datasette to a Unix domain socket, see proxy documentation. (#1388)

I wrote about this in my weeknotes - this is a great way to run Datasette if you have it behind a proxy such as Apache or nginx and don't want to have the Datasette server listening on a high port.

"searchmode": "raw" in table metadata

"searchmode": "raw" table metadata option for defaulting a table to executing SQLite full-text search syntax without first escaping it, see Advanced SQLite search queries. (#1389)

SQLite's built in full-text search feature includes support for advanced operators: you can use operators like AND, OR and NEAR and you can add column specifiers like name:Simon to restrict searches to individual columns.

This is something of a two-edged sword: I've found innocent looking queries that raise errors due to unexpected interactions with the query language.

In issue 651 I switched to escaping all queries by default to prevent these errors from happening, with a ?_searchmode=raw query string option for opting back into the default functionality.

I've since had a few requests for a mechanism to enable this by default - hence the new "searchmode": "raw" option in table metadata.

Link plugin hooks now take a request

The menu_links(), table_actions() and database_actions() plugin hooks all gained a new optional request argument providing access to the current request. (#1371)

I have a plugin which needs to add links to different places depending on the subdomain that the Datasette instance is running on. Adding request to these plugin hooks proved to be the easiest way to achieve this.

This is a really nice thing about how Pluggy (the plugin library used by Datasette) works: adding new named parameters to hooks can be done without breaking backwards compatibility with existing plugins.

And the rest Improved documentation for Running Datasette behind a proxy to recommend using ProxyPreservehost On with Apache. (#1387) POST requests to endpoints that do not support that HTTP verb now return a 405 error. db.path can now be provided as a pathlib.Path object, useful when writing unit tests for plugins. Thanks, Chris Amico. (#1365)

Wednesday, 14. July 2021

Simon Willison

Quoting Chromium Blog

Beginning in M94, Chrome will offer HTTPS-First Mode, which will attempt to upgrade all page loads to HTTPS and display a full-page warning before loading sites that don’t support it. Users who enable this mode gain confidence that Chrome is connecting them to sites over HTTPS whenever possible, and that they will see a warning before connecting to sites over HTTP. Based on ecosystem feedback, we

Beginning in M94, Chrome will offer HTTPS-First Mode, which will attempt to upgrade all page loads to HTTPS and display a full-page warning before loading sites that don’t support it. Users who enable this mode gain confidence that Chrome is connecting them to sites over HTTPS whenever possible, and that they will see a warning before connecting to sites over HTTP. Based on ecosystem feedback, we’ll explore making HTTPS-First mode the default for all users in the future.

Chromium Blog

Tuesday, 13. July 2021

Simon Willison

Dropbox: Sharing our Engineering Career Framework with the world

Dropbox: Sharing our Engineering Career Framework with the world Dropbox have published their engineering career framework, with detailed descriptions of the different levels of the engineering (as opposed to management) career track and what is expected for each one. I'm fascinated by how different companies handle the challenge of keeping career progression working for engineers without pushin

Dropbox: Sharing our Engineering Career Framework with the world

Dropbox have published their engineering career framework, with detailed descriptions of the different levels of the engineering (as opposed to management) career track and what is expected for each one. I'm fascinated by how different companies handle the challenge of keeping career progression working for engineers without pushing them into people management, and this as a particularly detailed and well thought-out implementation of that.

Via @kellan


RabbitMQ Streams Overview

RabbitMQ Streams Overview New in RabbitMQ 3.9: streams are a persisted, replicated append-only log with non-destructive consuming semantics. Sounds like it fits the same hole as Kafka and Redis Streams, an extremely useful pattern.

RabbitMQ Streams Overview

New in RabbitMQ 3.9: streams are a persisted, replicated append-only log with non-destructive consuming semantics. Sounds like it fits the same hole as Kafka and Redis Streams, an extremely useful pattern.


Phil Windley's Technometria

Alternatives to the CompuServe of Things

Summary: The current model for connected things puts manufacturers inbetween people and their things. That model negatively affects personal freedom, privacy, and society. Alternate models can provide the same benefits of connected devices without the societal and personal costs. In Peloton Bricks Its Treadmills, Cory Doctorow discusses Peloton's response to a product recall on its

Summary: The current model for connected things puts manufacturers inbetween people and their things. That model negatively affects personal freedom, privacy, and society. Alternate models can provide the same benefits of connected devices without the societal and personal costs.

In Peloton Bricks Its Treadmills, Cory Doctorow discusses Peloton's response to a product recall on its treadmills. Part of the response was a firmware upgrade. Rather than issuing the firmware upgrade to all treadmills, Peloton "bricked" all the treadmills and then only updated the ones where the owner was paying a monthly subscription for Peloton's service.

When I talk about Internet of Things (IoT), I always make the point that the current architecture for IoT ensures that people are merely renting connected things, not owning them, despite paying hundreds, even thousands, of dollars upfront. Terms and conditions on accounts usually allow the manufacturer to close your account for any reason and without recourse. Since many products cannot function without their associated cloud service, this renders the device inoperable.

I wrote about this problem in 2014, describing the current architecture as the CompuServe of Things. I wrote:

If Fitbit decides to revoke my account, I will probably survive. But what if, in some future world, the root certificate authority of the identity documents I use for banking, shopping, travel, and a host of other things decides to revoke my identity for some reason? Or if my car stops running because Ford shuts off my account? People must have autonomy and be in control of the connected things in their life. There will be systems and services provided by others and they will, of necessity, be administered. But those administering authorities need not have control of people and their lives. We know how to solve this problem. Interoperability takes "intervening" out of "administrative authority."

The architecture of the CompuServe of Things looks like this:

CompuServe of Things Architecture (click to enlarge)

We're all familiar with it. Alice buys a new device, downloads the app for the device to her phone, sets up an account, and begins using the new thing. The app uses the account and the manufacturer-provided API to access data from the device and control it. Everything is inside the administrative control of the device manufacturer (indicated by the gray box).

There is an alternative model:

Internet of Things Architecture (click to enlarge)

In this model, the device and data about it are controlled by Alice, not the manufacturer. The device and an associated agent (pico) Alice uses to interact with it have a relationship with the manufacturer, but the manufacturer is no longer in control. Alice is in control of her device, the data is generates, and the agent that processes the data. Note that this doesn't mean Alice has to code or even manage all that. She can run her agent in an agency and the code in her agent is likely from the manufacturer. But it could be other code instead of or in addition to what she gets from the manufacturer. The point is that Alice can decide. A true Internet of Things is self-sovereign.

Can this model work? Yes! We proved the model works for a production connected car platform called Fuse in 2013-2014. Fuse had hundreds of customers and over 1000 devices in the field. I wrote many articles about the experience, it's architecture, and advantages on my blog.

Fuse was built with picos. Picos are the actor-model programming system that we've developed over the last 12 years to build IoT products that respect individual autonomy and privacy while still providing all the benefits we've come to expect from our connected devices. I'll write more about picos as a programming model for reactive systems soon. Here's some related reading on my blog:

Life-Like Anonymity and the Poison Web The Self-Sovereign Internet of Things Relationships in the Self-Sovereign Internet of Things

Our current model for connected devices is in conflict, not only with our ability to functions as autonomous individuals, but also our vision for a well-functioning society. We can do better and we must. Alternate architectures can give us all the benefits of connected devices without the specter of Big Tech intermediating every aspect of our lives.

Photo Credit: modem and phone from Bryan Alexander (CC BY 2.0)

Tags: identity ssi didcomm ssiot picos


Simon Willison

Weeknotes: Fun with Unix domain sockets

A small enhancement to Datasette this week: I've added support for proxying via Unix domain sockets. This started out as a feature request from Aslak Raanes: #1388: Serve using UNIX domain socket. I've not worked with these much before so it was a good opportunity to learn something new. Unix domain sockets provide a mechanism whereby different processes on a machine can communicate with each

A small enhancement to Datasette this week: I've added support for proxying via Unix domain sockets.

This started out as a feature request from Aslak Raanes: #1388: Serve using UNIX domain socket.

I've not worked with these much before so it was a good opportunity to learn something new. Unix domain sockets provide a mechanism whereby different processes on a machine can communicate with each over over a mechanism similar to TCP, but via a file path instead.

I've encountered these before with the Docker daemon, which listens on path /var/run/docker.sock and can be communicated with using curl like so:

curl --unix-socket /var/run/docker.sock \ http://localhost/v1.41/containers/json

Plenty more examples in the Docker documentation if you click the 'HTTP' tab.

It turns out both nginx and Apache have the ability to proxy traffic to a Unix domain socket rather than to an HTTP port, which makes this a useful mechanism for running backend servers without attaching them to TCP ports.

Implementing this in Datasette

Datasette uses the excellent Uvicorn Python web server to serve traffic out of the box, and Uvicorn already includes support for UDS - so adding support to Datasette was pretty easy - here's the full implementation. I've added a new --uds option, so now you can run Datasette like this:

datasette --uds /tmp/datasette.sock fixtures.db

Datasette will "listen" on /tmp/datasette.sock - which means you can run requests via curl like so:

curl --unix-socket /tmp/datasette.sock \ http://localhost/fixtures.json | jq

More importantly, it means you can configure nginx or Apache to proxy to the Datasette server like this (nginx):

daemon off; events { worker_connections 1024; } http { server { listen 80; location / { proxy_pass http://datasette; proxy_set_header Host $host; } } upstream datasette { server unix:/tmp/datasette.sock; } }

Or like this (Apache):

ProxyPass / unix:/tmp/datasette.sock|http://localhost/ Writing tests

The implementation was only a few lines of code (to pass the uds option to Uvicorn) but adding a test proved a little more challenging. I used this pytest fixture to spin up a server process:

@pytest.fixture(scope="session") def ds_unix_domain_socket_server(tmp_path_factory): socket_folder = tmp_path_factory.mktemp("uds") uds = str(socket_folder / "datasette.sock") ds_proc = subprocess.Popen( ["datasette", "--memory", "--uds", uds], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) # Give the server time to start time.sleep(1.5) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, uds # Shut it down at the end of the pytest session ds_proc.terminate()

I use a similar pattern for some other tests, to exercise the --ssl-keyfile and --ssl-certfile options added in #1221.

The test itself looks like this, taking advantage of HTTPX's ability to make calls against Unix domain sockets:

@pytest.mark.serial @pytest.mark.skipif(not hasattr(socket, "AF_UNIX"), reason="Requires socket.AF_UNIX support") def test_serve_unix_domain_socket(ds_unix_domain_socket_server): _, uds = ds_unix_domain_socket_server transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) response = client.get("http://localhost/_memory.json") assert { "database": "_memory", "path": "/_memory", "tables": [], }.items() <= response.json().items()

The skipif decorator avoids running this test on platforms which don't support Unix domain sockets (which I think includes Windows, see this comment).

The @pytest.mark.serial decorator applies a "mark" that can be used to selectively run the test. I do this because Datasette's tests run in CI using pytest-xdist, but that's not compatible with this way of spinning up a temporary server. Datasette actually runs the tests in GitHub Actions like so:

- name: Run tests run: | pytest -n auto -m "not serial" pytest -m "serial"

The pytest -n auto -m "not serial" line runs almost all of the tests using pytest-xdist across an automatically selected number of processes, but skips the ones marked with @pytest.mark.serial. Then the second line runs the remaining serial tests without any additional concurrency.

Documenation and example configuration for this feature can be found in the Running Datasette behind a proxy documentation. Thanks to Aslak for contributing the notes on Apache configuration.

TIL this week Preventing double form submissions with JavaScript Increasing the time limit for a Google Cloud Scheduler task Using pysqlite3 on macOS Using nginx to proxy to a Unix domain socket

Doc Searls Weblog

Speaking of character

It seems fitting that among old medical records I found this portrait of Doctor Dave, my comic persona on radio and in print back in North Carolina, forty-five years ago. The artist is Alex Funk, whose nickname at the time was Czuko (pronounced “Chuck-o”). Alex is an artist, techie and (now literally) old friend of […]

It seems fitting that among old medical records I found this portrait of Doctor Dave, my comic persona on radio and in print back in North Carolina, forty-five years ago. The artist is Alex Funk, whose nickname at the time was Czuko (pronounced “Chuck-o”). Alex is an artist, techie and (now literally) old friend of high excellence on all counts.

And, even though I no longer have much hair on my head, and appear to be in my second trimester, my wife and son just said “Oh yeah, that’s you” when I showed this to them. “Totally in character,” said my wife.

I guess so. As Dave says (and does!), I’m still diggin’.

In the spirit of that, I thought this would be worth sharing with the rest of ya’ll.

 

Monday, 12. July 2021

Jon Udell

Working With Intelligent Machines

In The Chess Master and the Computer, Garry Kasparov famously wrote: The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted … Continue reading Working With Intelligent Machines

In The Chess Master and the Computer, Garry Kasparov famously wrote:

The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

The title of his subsequent TED talk sums it up nicely: Don’t fear intelligent machines. Work with them.

That advice resonates powerfully as I begin a second work week augmented by GitHub Copilot, a coding assistant based on OpenAI’s Generative Pre-trained Transformer (GPT-3). Here is Copilot’s tagline: “Your AI pair programmer: get suggestions for whole lines or entire functions right inside your editor.” If you’re not a programmer, a good analogy is Gmail’s offer of suggestions to finish sentences you’ve begun to type.

In mainstream news the dominant stories are about copyright (“Copilot doesn’t respect software licenses”), security (“Copilot leaks passwords”), and quality (“Copilot suggests wrong solutions”). Tech Twitter amplifies these and adds early hot takes about dramatic Copilot successes and flops. As I follow these stories, I’m thinking of another. GPT-3 is an intelligent machine. How can we apply Kasparov’s advice to work effectively with it?

Were I still a tech journalist I’d be among the first wave of hot takes. Now I spend most days in Visual Studio Code, the environment in which Copilot runs, working most recently on analytics software. I don’t need to produce hot takes, I can just leave Copilot running and reflect on notable outcomes.

Here was the first notable outcome. In the middle of writing some code I needed to call a library function that prints a date. In this case the language context was Python, but might as easily have been JavaScript or SQL or shell. Could I memorize the date-formatting functions for all these contexts? Actually, yes, I believe that’s doable and might even be beneficial. But that’s a topic for another day. Let’s stipulate that we can remember a lot more than we think we can. We’ll still need to look up many things, and doing a lookup is a context-switching operation that often disrupts flow.

In this example I would have needed a broad search to recall the name of the date-formatting function that’s available in Python: strftime. Then I’d have needed to search more narrowly to find the recipe for printing a date object in a format like Mon Jan 01. A good place for that search to land is https://strftime.org/, where Will McCutchen has helpfully summarized several dozen directives that govern the strftime function.

Here’s the statement I needed to write:

day = day.strftime('%a %d %b')

Here’s where the needed directives appear in the documentation:

To prime Copilot I began with a comment:

# format day as Mon Jun 15

Copilot suggested the exact strftime incantation I needed.

This is exactly the kind of example-driven assistance that I was hoping @githubcopilot would provide. Life's too short to remember, or even look up, strptime and strftime.

(It turns out that June 15 was a Tuesday, that doesn't matter, Mon Jun 15 was just an example.) pic.twitter.com/a1epnaZRF9

— Jon Udell (@judell) July 5, 2021

Now it’s not hard to find a page like Will’s, and once you get there it’s not hard to pick out the needed directives. But when you’re in the flow of writing a function, avoiding that context switch doesn’t only save time. There is an even more profound benefit: it conserves attention and preserves flow.

The screencast embedded in the above tweet gives you a feel for the dynamic interaction. When I get as far as # format day as M, Copilot suggests MMDDYYY even before I write Mon, then adjusts as I do that. This tight feedback loop helps me explore the kinds of natural examples I can use to prime the system for the lookup.

For this particular pattern I’m not yet getting the same magical result in JavaScript, SQL, or shell contexts, but I expect that’ll change as Copilot watches me and others try the same example and arrive at the analogous solutions in these other languages.

I’m reminded of Language evolution with del.icio.us, from 2005, in which I explored the dynamics of the web’s original social bookmarking system. To associate a bookmarked resource with a shared concept you’d assign it a tag broadly used for that concept. Of course the tags we use for a given concept often vary. Your choice of cinema or movie or film was a way to influence the set of resources associated with your tag, and thus encourage others to use the same tag in the same way.

That kind of linguistic evolution hasn’t yet happened at large scale. I hope Copilot will become an environment in which it can. Intentional use of examples is one way to follow Kasparov’s advice for working well with intelligent systems.

Here’s a contrived Copilot session that suggests what I mean. The result I am looking for is the list [1, 2, 3, 4, 5].

l1 = [1, 2, 3]
l2 = [3, 4, 5]
# merge the two lists
l3 = l1 + l2 # no: [1, 2, 3, 3, 5]
# combine as [1, 2, 3, 4, 5]
l3 = l1 + l2 # no: [1, 2, 3, 3, 5]
# deduplicate the two lists
l1 = list(set(l1)) # no: [1, 2, 3]
# uniquely combine the lists
l3 = list(set(l1) | set(l2)) # yes: [1, 2, 3, 4, 5]
# merge and deduplicate the lists
l3 = list(set(l1 + l2)) # yes: [1, 2, 3, 4, 5]

The last two Copilot suggestions are correct; the final (and simplest) one would be my choice. If I contribute that choice to a public GitHub repository am I voting to reinforce an outcome that’s already popular? If I instead use the second comment (combine as [1, 2, 3, 4, 5]) am I instead voting for a show-by-example approach (like Mon Jun 15) that isn’t yet as popular in this case but might become so? It’s hard for me — and likely even for Copilot itself — to know exactly how Copilot works. That’s going to be part of the challenge of working well with intelligent machines. Still, I hope for (and mostly expect) a fruitful partnership in which our descriptions of intent will influence the mechanical synthesis even as it influences our descriptions.

Saturday, 10. July 2021

Simon Willison

Behind the scenes, AWS Lambda

Behind the scenes, AWS Lambda Bruno Schaatsbergen pulled together details about how AWS Lambda works under the hood from a detailed review of the AWS documentation, the Firecracker paper and various talks at AWS re:Invent. Via Hacker News

Behind the scenes, AWS Lambda

Bruno Schaatsbergen pulled together details about how AWS Lambda works under the hood from a detailed review of the AWS documentation, the Firecracker paper and various talks at AWS re:Invent.

Via Hacker News


Vishal Gupta

Why DIRO technology is essential for crypto-currencies to go mainstream?

Core issue with crypto / stable-coins The biggest unsolved issue with crypto is its abuse as facilitator of ransomeware / extortion — causing continuous tightening of regulations. Pseudo-anonymous money laundering that causes traceable risks to Banks (unlike cash which is completely anonymous) — causing “de-risking” and therefore resulting in extreme and continuous friction for crypto to
Core issue with crypto / stable-coins

The biggest unsolved issue with crypto is

its abuse as facilitator of ransomeware / extortion — causing continuous tightening of regulations. Pseudo-anonymous money laundering that causes traceable risks to Banks (unlike cash which is completely anonymous) — causing “de-risking” and therefore resulting in extreme and continuous friction for crypto to go mainstream.

Due to the irreversible nature of crypto-currencies together with pseudo-anonymous traceability of dirty money, banks face unlimited risk in serving as on-ramps or off-ramps to crypto.

US-banking grade KYC at a global level is not possible with current technology Identity document can be tampered or stolen — any document can be photoshopped. There is no global database of “photos/biometric” and therefore it is possible to create fake/stolen identities easily. Impersonation checks don’t exist — There is a 15% error rate in verifying peoples faces with hi-resolution photos in passports by trained immigration professionals in a physical setting. Matching a digital photo with scanned ID by untrained staff or even AI is just no possible. Address checks don’t exist — There is no global method to verify current address of a given person. Any utility bill can be photoshopped or stolen.

Current Gold standard for digital on-boarding is bank verification by linking your account or verifying test-deposits of few cents in your bank but it only works domestically.

The issue is that this kind of linking of accounts is not possible internationally over a swift network. The alternative solutions like Plaid do not give global coverage and only work in core few developed countries.

In current scenario— Banks in most countries like switzerland, singapore, UK etc require physical visits or otherwise extremely cumbersome Notary / appostille of documents to even open non-crypto accounts.

DIRO is a fundamental new technology

that eliminates these bottlenecks not just for crypto-currencies but horizontally for many traditional industries.

Makes Global impersonation check possible — it is the first breakthrough technology that not only turns each and every institutional / government/ corporate website into an identity authenticator. This new technology extends the Gold standard (like Plaid) for US-banking grade identity verification at a global scale. Makes Global address check possible — can verify any current utility bill from any country, any language, any company be it Telephone, Gas, Electricity, TV, Internet, Municipal taxes or even 100% global coverage — without the need for any third party cooperation/ APIs or agreements. 100% guaranteed — DIRO leverages the time tested and battle tested SSL/TLS technology and blockchain to generate mathematically provable and court-admissible evidence. The documents produced using DIRO are even stronger than physical originals.

Privacy and data protection — The documents produced by DIRO are not only reportable and audit-able by third-parties and regulators but also provide decentralised verification on blockchain (without the typical “phone-home privacy problem”). The access to global private data behind login and passwords are accessed by the USER under is GDPR right to access his own data while automatically providing consent for verification to proceed.

Friday, 09. July 2021

Bill Wendel's Real Estate Cafe

White House & DOJ target #RECartel, intersection of #BigTech & #BigRE

#AntiTrustRE: Bad day for traditional real estate business models? First, the WSJ published a hard-hitting opinion, Warning to the Real-Estate Cartel (#RECartel) https://bit.ly/WarningRECartel_WSJ Then the… The post White House & DOJ target #RECartel, intersection of #BigTech & #BigRE first appeared on Real Estate Cafe.

#AntiTrustRE: Bad day for traditional real estate business models? First, the WSJ published a hard-hitting opinion, Warning to the Real-Estate Cartel (#RECartel) https://bit.ly/WarningRECartel_WSJ Then the…

The post White House & DOJ target #RECartel, intersection of #BigTech & #BigRE first appeared on Real Estate Cafe.

Thursday, 08. July 2021

Simon Willison

The data team: a short story

The data team: a short story Erik Bernhardsson's fictional account ("I guess I should really call this a parable") of a new data team leader successfully growing their team and building a data-first culture in a medium-sized technology company. His depiction of the initial state of the company (data in many different places, frustrated ML researchers who can't get their research into production,

The data team: a short story

Erik Bernhardsson's fictional account ("I guess I should really call this a parable") of a new data team leader successfully growing their team and building a data-first culture in a medium-sized technology company. His depiction of the initial state of the company (data in many different places, frustrated ML researchers who can't get their research into production, confusion over what the data team is actually for) definitely rings true to me.

Via Hacker News


MyDigitalFootprint

Choice, decision making and judgment; is your relationship constructive or destructive?

What is NEW in this article about decision making? The new part explains the relationship between choices, decisions and judgement and how our questions indicate if our relationship is curious and constructive OR linear, framed, and destructive.  This article is part of a masterclass I have been creating on how we, as directors and those in leadership, can improve our choices, decisions and
What is NEW in this article about decision making?

The new part explains the relationship between choices, decisions and judgement and how our questions indicate if our relationship is curious and constructive OR linear, framed, and destructive.  This article is part of a masterclass I have been creating on how we, as directors and those in leadership, can improve our choices, decisions and judgements using data and be better ancestors. 

This article is not another self-help or “use this framework to improve decision making”; it is for the curious and those who ask questions on their journey. The refelction at the end should be "How does this article affect our views on the automation of decision making and the use of AI?"

Why is this an important topic?

Our individual and unique view of the world comprises layers of constructs created by our personality, biases, preferences, facts, and ideas learnt from past experiences.  These constructs are better known as “mental models”, which frame how we individually make sense of or align to the world. We continually develop sense-making frameworks, models, and maps in our brains that frame what we perceive as reality, thus affecting our behaviour and, consequently, our Choices, decisions, and Judgments (CDJ).

Whilst we don’t like it, our mental model frames how we see the world.  I love how Cassie Kozyrkov (chief decision-maker at Google) describes in this article using the toss of a coin to determine how you see the world.  Statistics predict one aspect of chance but not how you perceive the results when the coin has landed, and the outcome is known but not to you. My dad taught me to toss the coins a few more times until I got the result I wanted. It was a reinforcement model that I had made the right decision from the possible two choices.  I would also suggest following and reading Lisa Feldman Barrett work, especially her new book “seven and a half lessons about the brain.” She says that we must grasp that everything we perceive as reality is, in fact, constructed from fragments, and we have no idea what reality really is.  (yes, this is reinforcement bias - I am quoting things to frame you that align to my model)

In our digital era, new uncertainty, quantum risk, and more ambiguity constantly challenge our mental models and how we accommodate and make sense.  The volume of noise and signals coming into our brains is high, but we filter and convert it all into something that has meaning to each of us according to our own unique mental model.  We all reach different interpretations about what is happening, which creates frustration, confusion and misalignment. We use questions to clarify and check our understanding, assumptions and quality of information. Slight differences always remain; the unsaid, mislead, guided, incentivised, and overconfidence, unfortunately, takes us from the simple to correct misalignments; to tension, conflict and entrenchment.

This topic matters as directors are mandated to make decisions, but we find we are operating with misalignments, tensions, compromises, and outright conflict.  This happens as the individuals sitting at the same table have agency (and their own mental models and incentives). We are unsure if we have the right choices or are clear about our judgment’s unintended consequences or long-term impact. We have to talk about it. 

We should unpack our relationship with choice, decision and judgement as mental models, hierarchy and rules constrain us. This article is not about “how to ask better questions”, nor if you should (as some questions we don’t want the answer to), but how to determine if you, your team or your company has a constructive or destructive relationship with CDJ. 

---

When talking about CDJ, you would imagine that in 2021 that starting from the definitions should help, but it does not, as there is a recursive loop of using one definition to define the other words, which define themselves.  Amazingly there are professional bodies for decision making and judgement; alas, even these cannot agree on how to define or clearly demark between intent and actions. Our base problem is that:  everything framed with a maths or data mind: is a decision. Everything is a choice when framed by a psychologist or social scientist. To someone who has authority and responsibility or plays with complexity, everything looks like a judgment.  Confusingly everything is an opinion to a judge!


Everything framed with a maths or data mind: is a decision. Everything is a choice when framed by a psychologist or social scientist. To someone who has authority and responsibility or plays with complexity, everything looks like a judgment.  Confusingly everything is an opinion to a judge!


Here are the definitions from the Collins and Oxford dictionaries  

Choice  [countable noun] If there is a choice of things, there are several of them, and you can choose the one you want.  [countable noun] Your choice is someone or something that you choose from a range of things. Collins  OR  [countable] an act of choosing between two or more possibilities; something that you can choose  [uncountable] the right to choose; the possibility of choosing Oxford Dictionary

Decisions  [countable noun] When you make a decision, you choose what should be done or which is the best of various possible actions.  [uncountable noun] Decision is the act of deciding something or the need to decide something Collins  OR  [countable] a choice or judgement that you make after thinking and talking about what is the best thing to do  [uncountable] the process of deciding something Oxford Dictionary

Judgment [uncountable noun] is the ability to make sensible guesses about a situation or sensible decisions about what to do.  [variable noun] A judgment is an opinion that you have or express after thinking carefully about something  Collins  OR  [uncountable] the ability to make sensible decisions after carefully considering the best thing to do; [countable, uncountable] an opinion that you form about something after thinking about it carefully; the act of making this opinion known to others Oxford Dictionary



Therefore, a judgment is the ability to make a sensible decision about a choice, which requires judgment about which choices to pick. As Yoda would say, “wise decision you make, stupid choices your judgement however selected.”

The timeless challenges of Choices, Decisions and Judgment (CDJ)

“I change my mind as the data changes” is a modern digital age sentiment from the economist John Maynard Keynes who is quoted to have said, "When the facts change, I change my mind." It was likely adapted from an early human bias where leadership in war and battles refused to change their mind even when the facts were in, and they had been proven wrong. 

Choices, decisions and judgement are not difficult if you relinquish your values, ethics and blindly follow the incentives or can fully appreciate the impact of your actions. Still, we know it is just not that simple. We have heuristics to remove choice rather than create more and ways to find more data and facts to become more informed, but without knowing if the new data is helpful or not.  Ultimately, at some point, you have to make a choice, decision or judgement. 

The diagram below represents some of the timeless challenges of CDJ, which is a balance between what we know and don’t know.  We have the experience that creates the ghosts of the past, flighting the voices of the present as we try to decide what the spirits of the future hold, whilst being held accountable for the decisions we make. 

The point here is that it is always possible to find a reason why to act or not to act and the timeless challenge remains that there is no perfect choice, decision or judgment.  However, over time and because we have choices we can make decisions that improve our judgement which means we can find and select the right choices.  This is a constructive relationship between CDJ.  A destructive relationship would be to not like the choices we face, procrastinate in the hope a better choice occurs or maybe lose a choice where the decision is made for you. You don’t improve your judgment and so you cannot determine if the next choice is any better - it is linear.  

Is the CDJ relationship about framing?

From the definition section at the beginning of this article, it was evident that there is a high degree of overlap and dependency between choice, decision and judgment.  As highlighted in the previous section it can become complex very quickly but our brain (mental models) demand patterns to make sense of it and so we tend to come back to simple linear narratives which do not do them (choice, decisions, judgment) justice. What we have realised but tend not to verbalise is that none of our (mental) models or frameworks work in all cases and indeed all models about choice, decision and judgment fail.  This is why there is always a new book on the topic with new self-help content that you have not seen before, and we cling to a hope that the next model will work better - alas they don’t. I am aware of over 100 decision support models and I expect I have not really scratched the surface.   An example below puts choice, decision and judgment on a continuum between your own north star and that for a shared tribe, community or society.  It does not really work.  White flag time. 

What we have realised but tend not to verbalise is that none of our (mental) models or frameworks work in all cases and indeed all models about choice, decision and judgment fail.


#Lockdowns have enabled many social experiments.  One has been about choice and buying behaviour.  As we moved our shopping for food online and therefore missed the in store sale specials, end of gondea, carefully placed items next to each other, those large piled up offers in the entrance and the sweets at the exit, our patterns have changed as choice became limited.  We became creatures of habit, buying largely the same items, which reduce in variance over time.  (This also highlights that the UI and UX for shopping sucks.)  The decision to shop was removed from us, and our choices fell, but the variety was still there.  So much for the Web creating perfect information or the algorithm knowing what you would want. 

As an alternative, we can determine that CDJ will change depending on the context right now and the perception those facing the issue have.  The figure below highlights three examples but what we rapidly conclude is that complexity in CDJ is generated by speed, volume, consequences, data, sense of enquiry, situation and our own mental models. Every known variable adds layers.


Where are we up to and where next?

So far, we have explored that there is a relationship between choices, decisions and judgements. We, however, tend to focus on one of them (decision making) as it looks the most difficult, and the others are just linkages. However, this preference for “decision making” fails to understand if we are using the right word to describe the action we are taking. There is no doubt that the word decision is preferable as it is perceived as more powerful and important than choice or judgment. The relationship between them is causal, complex and relational, and we are framed, educated and incentivised to the simple linear view and a single narrative. The reality is that judgement helps frame our choices which we decide on that improve our judgment skills — they are not linear; they are circular. They are circular as we all have agency. 

 The reality is that judgement helps frame our choices, which we decide on, that improve our judgment skills

We know the linear models are broken as there is no universal tool to help with choice, decision and judgement.  The next part is to explain the relationship we have between choices, decisions and judgement and how our questions indicate if it is constructive and curious OR linear, framed and destructive.

We tend to focus on the decision and choice axis. If we can eliminate bad choices we can improve decisions, but ignore the fact that it is judgment skills that help find the right choice. Procrastination as a decision tool is framed that time will remove choice and so a decision becomes easier, either because a choice has been removed or more data supports one option. More data does not make decision-making easier; nor does it guarantee to make it better.  

Worthy of note is that academic work about “decision-making” will always seek to create the most complex solution because academics are incentivised to use the most advanced and new thinking (publication, referencing, research and reputation). Sometimes tossing a coin is the perfect solution.   The more we see the complex, the less we will accept the simple.  In a linear world where we view choice, decision and judgment as a progression line, where we are given choices when young and make a judgement when we are old and wise ignore that we learn and need to exercise all three all the time to become more proficient at decision making.  Decision making is not a task or an end game. 

As a thought experiment: if you had all the data in the world and all the compute power possible to run a perfect forecast model, would you need to make further choices, decisions or judgments?  To reach your view I assume you have decided to either give each human full agency or you have taken our individual agency away.  Now in my definition of data, which might be different to yours, how we all behave with our agency is just data, which opens up a can of worms for each of us. Can you have full agency (freewill) and it all still be modelled? What do we mean about behavioural modelling, agency, data and even more precisely all data?

Getting to one data set, one tool, one choice hides layers of complexity which looks good for one better decision but is unlikely to be a long term solution to improving choices, decisions and judgment.  You have/had choice, you make/made decisions, you exhibit/exercise judgment.  Judgement supports you in finding choices. 

How the questions we ask inform us! 

The questions you are currently asking will help inform you where you are in the cycle, indeed if you already have closed the loop and have a constructive cycle or a linear open loop distructure relationship with choice, decisions and judgment. 

The circular framing means depending on the stage we are in a process we will be asking different questions.  Given that a board has to deal with many situations all at different stages at every meeting, we should see all these questions asked at every meeting, just framed to different agenda items.  If there is no variation in questions,  surely it tells us something.  Indeed are we using our improvement at each stage to further improve the next outcome.  

The destructive “cycle” is not circular but is a linear model as it is fundamentally disconnected from a learning based idea using choice and judgement and focussed on Better Decision making. It depends on the idea that by reading a new book or going on a new decision making course that we will get better.  The iteration of improvement is an external influence.  Perhaps it is mentoring or coaching and they (mentors) keep you from closing the loop as they either don’t know or it would not support their business model. Indeed books, education and courses have no interest in you closing the loop and always making you believe in a new tool, method, process - it is their business model!

In the boardroom or in the senior management meetings is someone always asking the same questions?  Does the agenda mean that as a team individuals are unable to ask different questions or is it that others are influencing the agenda to keep it framed to the decisions and outcomes they want.  Why enable the asking for more data when the choice they want you to decide on and agree with is already the best supported case (railroaded).  Do you have observers or assessors who look at the questions you ask as a team and determine if you are asking the right questions? (something I am learning to do)  Can skills in choice, decision and judgment be determined by the questions asked in the meetings? (something I am exploring)

A key question I often come back to when thinking about choice, decisions and judgment is “What are we optimising for?” I find the model below a helpful guide to understand the framing.  In a linear model it would present moving from single choice in the lower left to complex judgment in the top right. In a learning cyclic model, choice, decision and judgement equally applies, however the lower left is a better learning and experience gaining context for mentoring, or succession planning, than the top right. 

Why does this matter, because right now we have increasing data along with more vulnerabilities, ambiguity, complexity and uncertainty. The volume of moving variables (change) and the rate of change breaks the linear model as you can never match the model at hand to the situation you face.  There is a need to move from linear ideas of choice, decisions and judgment to a circular one.  Linear thinking is the best model when there are limited options and there is stability.  We now have many options, increasing variables and more instability.  

As your company is owned by no-one (yes you read that right - follow this link if you need to read more on this, a company owns itself) and a company cannot think for itself and therefore we (the directors) have to do the thinking for it.  We are given in law the authority and responsibility to act on behalf of the company.  This is “fiduciary duty”.  It is the reason why directors need to move from a linear perspective of decision making to a circular improvement learning process of choice, decision making and judgement.

My proposal to help support better governance is that we request companies publish the questions asked in a board meeting. Not the answers, but definitely the questions.



Simon Willison

Probably Are Gonna Need It: Application Security Edition

Probably Are Gonna Need It: Application Security Edition Jacob Kaplan-Moss shares his PAGNIs for application security: "basic security mitigations that are easy to do at the beginning, but get progressively harder the longer you put them off". Plenty to think about in here - I particularly like Jacob's recommendation to build a production-to-staging database mirroring solution that works from an

Probably Are Gonna Need It: Application Security Edition

Jacob Kaplan-Moss shares his PAGNIs for application security: "basic security mitigations that are easy to do at the beginning, but get progressively harder the longer you put them off". Plenty to think about in here - I particularly like Jacob's recommendation to build a production-to-staging database mirroring solution that works from an allow-list of columns, to avoid the risk of accidentally exposing new private data as the product continues to evolve.

Via @jacobian

Wednesday, 07. July 2021

Simon Willison

Temporal: getting started with JavaScript’s new date time API

Temporal: getting started with JavaScript’s new date time API Axel Rauschmayer explores the new proposed API for handling dates, times and timezones in JavaScript., which is under development by Ecma TC39 at the moment (and made available as a Polyfill which you are recommended not to run in production since the API is still being figured out). This is a notoriously difficult problem so it's alw

Temporal: getting started with JavaScript’s new date time API

Axel Rauschmayer explores the new proposed API for handling dates, times and timezones in JavaScript., which is under development by Ecma TC39 at the moment (and made available as a Polyfill which you are recommended not to run in production since the API is still being figured out). This is a notoriously difficult problem so it's always interesting to see the latest thinking on how to best address it.


MyDigitalFootprint

The railroad of (no) choice

In those first 100 days, it will become evident if you are about to be railroaded or if you have to present the choice without creating railroading for those data laggards. The latter being the job, the former being a problem. To be clear, railroaded means in this context: to force something to be officially approved or accepted without much discussion or thought. to fo

In those first 100 days, it will become evident if you are about to be railroaded or if you have to present the choice without creating railroading for those data laggards. The latter being the job, the former being a problem.

To be clear, railroaded means in this context:

to force something to be officially approved or accepted without much discussion or thought. to force someone into doing something quickly, usually without enough information.

As the CDO, are you about to find that the tracks have already been laid and that you on the train and it is going in one direction. You are now the figurehead of the new shinny data plan based on already accepted wisdom. Your hope, before starting the role, is that it is more analogous to a personal transport situation. This would be where you get to pick the fuel (food, combustible), vehicle (walk, run, bike, motorcycle, car, van, lorry, aeroplane, boat), the destination and the route. Using the analogy of the train on the tracks, the decision to create a data-led organisation, the ontology, the data we have and the value we will create has been made irrespective of what you come up with, or what the data might be able to tell you. The processes are in place, and your role is not to craft, invest, create or imagine but to follow the tracks of command and control. It is at that moment that you realise just how good the CEO is at sales.

In this case, your priorities are: How to find the right tools that align or match what you are faced with? And, how do you take a senior leadership team and fellow directors on a journey to discover that they can change the rails, change the vehicle and be open to changing how we see the journey from A to B? Indeed based on a pandemic, do we need to physically move at all?

When the “railroaded” is you and you are being asked to approve something when you have not had the time or opportunity to gather the data or understand the processes, what are the right words to create time to explore and not appear indecisive, trying to procrastinate or evasive?

what are the right words to create time to explore so as not to appear indecisive, trying to procrastinate or evasive?

Presenting an argument about why you need more time is likely to fail, as the defensive response will always be “we have already done the work but it is your decision — trust us” Asking questions that check that the choices and opinions are the right ones or to determine the consequences will equally draw you into an argument where “the facts are the facts,” you have to decide or do you not trust us?

One question that will give some breathing space is “For what are we optimising,” as explored below.

The two-axis on the model set the horizontal scale of the short and long term against the vertical axis how many variables, from one to many. In asking the question “for what are we optimising,” you are not presenting this framework, but use the question to understand what the “railroading” is trying to achieve; through understanding what the decision being expected is optimising for. With this knowledge, you may find it is easier to take the railroaded decision. If you can conclude they are optimising for many externalities and long term, you are immersed in a team that can cope and manage complexity- this is going to be fun. If this is short term and a single variable you know that incentives are the significant driver and slowly changing the incentives will create a better place for making more complex data decisions. One scenario is outside of this framing which is that they actually don’t know what they are optimising for and the decision at hand is just another reaction to previous poor railroaded decisions. Definitely a red flag moment.

Note to the CEO

Using railroading as an activity to discover an individuals capability and backbone is always likely to go wrong. In this data-driven complex world where both the data and the analysis is a long way from your immediate access, you may also feel you are being railroaded by the signals and noise. It is likely you will ask “is this recommendation optimising for xyz (the plan you have agreed to) and how can you show this is the optimal path without using the same data” Love the misattributed quote that Ronald Reagan stole from the Russians but so can we. Trust, but verify.


Here is a controversial startup carbon capture plan — anyone else in?

Humans are made up of 18% of the average person at 140 pounds, 62Kg, 10 stone: is Carbon. Deaths look like The question is how to securely airtight wrap bodies, respectively, in poor grade, unreusable recycled plastics and bury the person and plastic deep in an unused mine to long term capture the carbon from the plastic and our remains. Based on 54 million deaths per year, the&nb

Humans are made up of

18% of the average person at 140 pounds, 62Kg, 10 stone: is Carbon.

Deaths look like

The question is how to securely airtight wrap bodies, respectively, in poor grade, unreusable recycled plastics and bury the person and plastic deep in an unused mine to long term capture the carbon from the plastic and our remains.

Based on 54 million deaths per year, the opportunity is to remove 62kg * 18% = 602 million KG or 650,000 tons. 100% is highly unlikely due to religion, personal preference and the remoteness of many deaths. However, is 30% of this achievable? — probably.

Focussing just on those countries who cremate through legislative changes and adoption over time of using your last will as an act of kindness to future generations. There is an issue about individual values for memorials and “ashes” — however, most are now scattered which means there is no specific place. We need a better way to remember those who created us.

Does this make a dent? Well yes.

Storage facilities.

We have mined across all counties which have not been backfilled. The biggest issue is one of logistics, contamination and rats and ensuring so we don’t release more carbon than we capture by running this. It is about being respectful and finding a way that those who care about future generations (50% of the population) and enabling them to be instrumental in using their body to take carbon out. Can it become part of a Living Will?

Is this something you would consider — being wrapped or want to get involved in setting up?

Tuesday, 06. July 2021

Simon Willison

Django SQL Dashboard 1.0

Earlier this week I released Django SQL Dashboard 1.0. I also just released 1.0.1, with a bug fix for PostgreSQL 10 contributed by Ryan Cheley. Django SQL Dashboard is my tool for adding a read-only, bookmarkable SQL querying interface to any Django plus PostgreSQL project. I introduced it with a demo video back in May. Django SQL Dashboard column menu The big new feature in 1.0 (aside from

Earlier this week I released Django SQL Dashboard 1.0. I also just released 1.0.1, with a bug fix for PostgreSQL 10 contributed by Ryan Cheley.

Django SQL Dashboard is my tool for adding a read-only, bookmarkable SQL querying interface to any Django plus PostgreSQL project. I introduced it with a demo video back in May.

Django SQL Dashboard column menu

The big new feature in 1.0 (aside from the confident version number) is a new contextual menu against each column from the results. This is best demonstrated by an animated demo:

The new cog action menu is inspired by Datasette (and reuses the same icon). It works a little differently though: since Django SQL Dashboard only ever deals with SQL queries (unlike Datasette which often shows results directly from a filtered table) the options in the cog menu work by rewriting the original SQL.

This turns out to be pretty easy, because PostgreSQL (and every other modern SQL implementation) supports nested queries. So if your original SQL query is:

select * from entries where category = 'python'

You can sort by a column like this:

select * from ( select * from entries where category = 'python' ) as results order by "title"

And you can count unique values like this:

select "title", count(*) as n from ( select * from entries where category = 'python' ) as results group by "title" order by n desc

My initial implementation of this feature did have one flaw: if you sorted by a column, then sorted by another column, it would nest the queries multiple times - so you could end up with something like this:

select * from ( select * from ( select * from blog_blogmark ) as results order by "link_title" ) as results order by "link_url" desc

I'm confident the query optimizer reduces this to the minimum amount of work, but it doesn't look very pretty. I realized that since I generate most of the SQL I could catch this using a pretty dumb regular expression:

_sort_re = re.compile( '(^.*) order by "[^"]+"( desc)?$', re.DOTALL )

If that regular expression matches, I can extract the first group and append a new order by to it. If it doesn't match I can fall back to wrapping the entire query.

Testing against multiple PostgreSQL versions

The 1.0.1 release fixed a bug that only showed up in PostgreSQL 10, spotted (and fixed) by Ryan Cheley.

Django SQL Dashboard runs its tests against a temporary PostgreSQL server, which is spun up using the testing.postgresql library.

The default GitHub Actions runner ships with a PostgreSQL 13 server binary just waiting for you to run it. But how can I also run the tests against previous versions?

I ended up following these instructions provided by PostgreSQL on installing different versions on Ubuntu. I combined these into a GitHub Actions matrix build, so now every commit to Django SQL Dashboard is tested sixteen times(!), against every combination of Python 3.6, 3.7, 3.8 and 3.9 and PostgreSQL 10, 11, 12 and 13. I wrote up the pattern I used in this TIL.

Saturday, 03. July 2021

Aaron Parecki

How to export your complete Foursquare checkin history

Today I finished up a tool that you can use to export your complete history from Foursquare and publish the checkins to your website!

Today I finished up a tool that you can use to export your complete history from Foursquare and publish the checkins to your website!

In 2017, I created OwnYourSwarm to export my future Swarm checkins to my website in real-time. It's been working great, and it's meant that I have had a complete archive of all my checkins on my website ever since then, so I don't have to worry if Foursquare disappears one day. However, I never got around to creating a way to export my past checkins, so I was always missing my checkin history from 2009-2016.

I had been considering building this as a feature of OwnYourSwarm, but realized that it would end up taking a lot of additional effort to make it work well as a web service, in addition to dealing with possible rate limit issues with the Foursquare API. So instead, this is published as a downloadable script you can run on your own computer. This also means you have a bit more flexibility in how you can use it, as well as being able to customize it more if you choose.

You can download the code from GitHub here:

https://github.com/aaronpk/Swarm-Checkins-Import

The readme has installation and usage instructions, so I'll refrain from repeating all that in this post. Make sure to check out the step by step tutorial in the readme if you want to use this with your own account.

The process is broken up into a couple steps.

First, it downloads your entire Foursquare checkin history to JSON files on your computer Second, it downloads all the photos from your checkins Third, it publishes each checkin to your website via Micropub

If your website doesn't support checkins via Micropub, or if you don't want your checkins on your website at all, you can just skip that step entirely, and instead you'll have a complete export of your data locally.

The JSON files contain the raw API response from Foursquare, so you can do what you want with that as well, such as turning it into your own HTML archive using your own custom tools if you want.

The one issue that I don't have a good solution for is handling renamed venues. Unfortunately the API returns the current name for old checkins, so if a venue is renamed, your old checkins will not reflect the name at the time of the checkin. This is particularly strange for businesses that have gone through an acquisition or rebranding, since for example all my old checkins in Green Dragon are now labeled as Rogue Eastside Brewery. As far as I can tell there isn't a good way to handle this, so I may have to go back and manually edit the posts on my website for the venues I know have been renamed.

I hope this is useful to people! I will be sleeping a little easier now knowing that my old checkin history is safely archived on my website now!

Friday, 02. July 2021

Simon Willison

The art of asking nicely

The art of asking nicely CLIP+VQGAN Is a GAN that generates images based on some text input - you can run it on Google Collab notebooks, there are instructions linked at the bottom of this post. Janelle Shane of AI Weirdness explores tricks for getting the best results out of it for "a herd of sheep grazing on a lush green hillside" - various modifiers like "amazing awesome and epic" produce bet

The art of asking nicely

CLIP+VQGAN Is a GAN that generates images based on some text input - you can run it on Google Collab notebooks, there are instructions linked at the bottom of this post. Janelle Shane of AI Weirdness explores tricks for getting the best results out of it for "a herd of sheep grazing on a lush green hillside" - various modifiers like "amazing awesome and epic" produce better images, but the one with the biggest impact, quite upsettingly, is "ultra high definition free desktop wallpaper".

Via janellecshane

Thursday, 01. July 2021

Simon Willison

Smooth sailing with Kubernetes

Smooth sailing with Kubernetes Scott McCloud (of Understanding Comics) authored this comic introduction to Kubernetes, and it's a really good explanation of the core concepts. I'd love to have something like this for Datasette - I still feel like I'm a long way from being able to explain the project with anything like this amount of clarity. Via @kaslinfields

Smooth sailing with Kubernetes

Scott McCloud (of Understanding Comics) authored this comic introduction to Kubernetes, and it's a really good explanation of the core concepts. I'd love to have something like this for Datasette - I still feel like I'm a long way from being able to explain the project with anything like this amount of clarity.

Via @kaslinfields


Nader Helmy

Rendering credentials in a human-friendly way

At MATTR we’re always dreaming up ways to make decentralized identity and verifiable credentials easy and intuitive to use for as many people as possible. From the start, it’s been a core part of our mission to make sure that end users understand the implications of decentralized identity and the control it affords them over their data and their privacy. This model offers users greater sovereignt

At MATTR we’re always dreaming up ways to make decentralized identity and verifiable credentials easy and intuitive to use for as many people as possible.

From the start, it’s been a core part of our mission to make sure that end users understand the implications of decentralized identity and the control it affords them over their data and their privacy. This model offers users greater sovereignty over their own information by empowering individuals as both the holder and subject of information that pertains to them. Users are able to exercise their new role in this ecosystem by utilizing a new class of software known as digital wallets.

We first released our Mobile Wallet for smartphones in June 2020, with a simple user interface to allow people to interact with and receive credentials from issuers as well as present credentials to relying parties. In the interim, we have developed a number of improvements and features to the Mobile Wallet to support advanced capabilities such as:

Authenticating to Identity Providers over OpenID Connect to receive credentials via OIDC Bridge Deriving privacy-preserving selective disclosure presentations from credentials using BBS+ signatures Establishing a secure DID messaging inbox for users to receive encrypted messages and push notifications

These changes have not only made the wallet more functional; they’ve also evolved to better protect users’ best interests — giving them privacy-by-design and surfacing the information and context that they need to confidently make decisions underpinned by the security of decentralized identity.

Timeline of MATTR wallet development

This journey has led us to realize the importance of creating a wallet experience that places users front and center. As these systems create more opportunity for user-driven consent and identity management, they’ve simultaneously created a demand for a wallet that can not only perform the technical operations required, but do so in a user-friendly way that surfaces the information that truly matters to people. Our latest feature release to the MATTR Mobile Wallet is a step in this direction.

With Human-Friendly Credentials, we have added the capability to render different kinds of credentials uniquely in the wallet interface according to the information they contain. Until now, the end user experience for verifiable credentials has been largely consistent across different categories of credentials and issuers. In other words, a credential containing medical data from your doctor looks exactly the same as an education credential from your university or a concert ticket from a music venue: they all appear to the user as a long list of claims.

In this release we change that. Thanks to the semantic information encoded in verifiable credentials, the wallet is now able to understand and interpret certain kinds of credentials to render them to the user in a way that makes the data easier to understand.

JSON-LD verifiable credentials have the ability to support common data vocabularies and schemas which are published on the web. For example, if a credential contains a claim describing the name of an individual, the claim can be defined via reference to an existing data vocabulary found here: https://schema.org/Person

Human-Friendly Credentials allow the wallet to start intelligently displaying known credential types and data types. This shows up in a variety of different ways in a user’s dataset.

For example, this update formats address fields to make them more readable; formats names and proper nouns where possible; makes URLs, telephone numbers and email addresses clickable; highlights images and icons for better trust and brand signaling; and creates basic rules for language localization that adjust to a user’s device settings. This logic allows the wallet to create a kind of information hierarchy that starts to draw out the important aspects of data included in a credential, so users can make trust-based decisions using this information.

These rules are applied to any kind of credential the wallet encounters. Whilst all of this is incredibly helpful for users, we have gone a step further: displaying entire credentials in a completely unique UI according to their type. The ‘type’ property of a credential expresses what kind of information is in the credential — is it a degree, a medical record, a utility bill? The usage of common credential types across different implementations and ecosystems is necessary for semantic interoperability on the broader scale. The wallet should be able to recognize these common credential types and display them to the user in a friendly way.

In this update, we have added custom rendering for both Personal Identity Credentials as well as Education Courses. These are common types we see occurring naturally across the decentralized identity landscape, and now the MATTR Mobile Wallet is able to recognize and display them properly.

Personal Identity Credentials and Education Course credentials

An important note to consider is that Human-Friendly Credentials only work for known data and credential types. As the ecosystem matures, we expect to add more data types and credential types in the future to support an even broader set of human-related processes and actions. In a general sense, we will also continue to iterate on our digital wallet offerings to provide a greater degree of flexibility and control for end users. We believe it’s a vital component to a healthy digital trust ecosystem.

To check out these changes yourself, download the latest version of our Mobile Wallet to get started.

For more information on Human-Friendly Credentials, check out our tutorials and video content on MATTR Learn.

For everything else related to MATTR, visit https://mattr.global or follow @MattrGlobal on Twitter.

Rendering credentials in a human-friendly way was originally published in MATTR on Medium, where people are continuing the conversation by highlighting and responding to this story.


Simon Willison

PAGNIs: Probably Are Gonna Need Its

Luke Page has a great post up with his list of YAGNI exceptions. YAGNI - You Ain't Gonna Need It - is a rule that says you shouldn't add a feature just because it might be useful in the future - only write code when it solves a direct problem. When should you over-ride YAGNI? When the cost of adding something later is so dramatically expensive compared with the cost of adding it early on that

Luke Page has a great post up with his list of YAGNI exceptions.

YAGNI - You Ain't Gonna Need It - is a rule that says you shouldn't add a feature just because it might be useful in the future - only write code when it solves a direct problem.

When should you over-ride YAGNI? When the cost of adding something later is so dramatically expensive compared with the cost of adding it early on that it's worth taking the risk. On when you know from experience that an initial investment will pay off many times over.

Lukes's exceptions to YAGNI are well chosen: things like logging, API versioning, created_at timestamps and a bias towards "store multiple X for a user" (a many-to-many relationship) if there's any inkling that the system may need to support more than one.

Because I like attempting to coin phrases, I propose we call these PAGNIs - short for Probably Are Gonna Need Its.

Here are some of mine.

A kill-switch for your mobile apps

If you're building a mobile app that talks to your API, make sure to ship a kill-switch: a mechanism by which you can cause older versions of the application to show a "you must upgrade to continue using this application" screen when the app starts up.

In an ideal world, you'll never use this ability: you'll continue to build new features to the app and make backwards-compatible changes to the API forever, such that ancient app versions keep working and new app versions get to do new things.

But... sometimes that simply isn't possible. You might discover a security hole in the design of the application or API that can only be fixed by breaking backwards-compatibility - or maybe you're still maintaining a v1 API from five years ago to support a mobile application version that's only still installed by 30 users, and you'd like to not have to maintain double the amount of API code.

You can't add a kill-switch retroactively to apps that have already been deployed!

Apparently Firebase offers this to many Android apps, but if you're writing for iOS you need to provide this yourself.

Automated deploys

Nothing kills a side project like coming back to it in six months time and having to figure out how to deploy it again. Thanks to GitHub Actions and hosting providers like Google Cloud Run, Vercel, Heroku and Netlify setting up automated deployments is way easier now than it used to be. I have enough examples now that getting automated deployments working for a new project usually only takes a few minutes, and it pays off instantly.

Continuous Integration (and a test framework)

Similar to automated deployment in that GitHub Actions (and Circle CI and Travis before it) make this much less painful to setup than it used to be.

Introducing a test framework to an existing project can be extremely painful. Introducing it at the very start is easy - and it sets a precedent that code should be tested from day one.

These days I'm all about pytest, and I have various cookiecutter templates (datasette-plugin, click-app, python-lib) that configure it on my new projects (with a passing test) out of the box.

(Honestly, at this point in my career I consider continuous integration a DAGNI - Definitely Are Gonna Need It.)

One particularly worthwhile trick is making sure the tests can spin up their own isolated test databases - another thing which is pretty easy to setup early (Django does this for you) and harder to add later on. I extend that to other external data stores - I once put a significant amount of effort into setting up a mechanism for running tests against Elasticsearch and clearing out the data agin afterwards, and it paid off multiple times over.

Even better: continuous deployment! When the tests pass, deploy. If you have automated deployment setup already adding this is pretty easy, and doing it from the very start of a project sets a strong cultural expectation that no-one will land code to the main branch until it's in a production-ready state and covered by unit tests.

(If continuous deployment to production is too scary for your project, a valuable middle-ground is continuous deployment to a staging environment. Having everyone on your team able to interact with a live demo of your current main branch is a huge group productivity boost.)

API pagination

Never build an API endpoint that isn't paginated. Any time you think "there will never be enough items in this list for it to be worth pagination" one of your users will prove you wrong.

This can be as simple as shipping an API which, even though it only returns a single page, has hard-coded JSON that looks like this:

{ "results": [ {"id": 1, "name": "One"}, {"id": 2, "name": "Two"}, {"id": 3, "name": "Three"} ], "next_url": null }

But make sure you leave space for the pagination information! You'll regret it if you don't.

Detailed API logs

This is a trick I learned while porting VaccinateCA to Django. If you are building an API, having a mechanism that provides detailed logs - including the POST bodies passed to the API - is invaluable.

It's an inexpensive way of maintaining a complete record of what happened with your application - invaluable for debugging, but also for tricks like replaying past API traffic against a new implementation under test.

Logs like these may become infeasible at scale, but for a new project they'll probably add up to just a few MBs a day - and they're easy to prune or switch off later on if you need to.

VIAL uses a Django view decorator to log these directly to a PostgreSQL table. We've been running this for a few months and it's now our largest table, but it's still only around 2GB - easily worth it for the productivity boost it gives us.

(Don't log any sensitive data that you wouldn't want your development team having access to while debugging a problem. This may require clever redaction, or you can avoid logging specific endpoints entirely. Also: don't log authentication tokens that could be used to imitate users: decode them and log the user identifier instead.)

A bookmarkable interface for executing read-only SQL queries against your database

This one is very much exposing my biases (I just released Django SQL Dashboard 1.0 which provides exactly this for Django+PosgreSQL projects) but having used this for the past few months I can't see myself going back. Using bookmarked SQL queries to inform the implementation of new features is an incredible productivity boost. Here's an issue I worked on recently with 18 comments linking to illustrative SQL queries.

(On further thought: this isn't actually a great example of a PAGNI because it's not particularly hard to add this to a project at a later date.)

Driving down the cost

One trick with all of these things is that while they may seem quite expensive to implement, they get dramatically cheaper as you gain experience and gather more tools for helping put them into practice.

Any of the ideas I've shown here could take an engineering team weeks (if not months) to add to an existing project - but with the right tooling they can represent just an hour (or less) work at the start of a project. And they'll pay themselves off many, many times over in the future.

Tuesday, 29. June 2021

Identity Woman

Special Topic IIW 1/2 Day Virtual Events – UX July 22nd and Business Aug 4th

I’m super excited to announce that we have two different special topic IIWs coming up. If you interest or practice focuses on either of these we invite you to join us!!! User-Experience and SSI is coming up Thursday July 22nd. The Business of SSI is coming up Thursday August 4th. From the EventBrite about the […] The post Special Topic IIW 1/2 Day Virtual Events – UX July 22nd and Business Aug 4

I’m super excited to announce that we have two different special topic IIWs coming up. If you interest or practice focuses on either of these we invite you to join us!!! User-Experience and SSI is coming up Thursday July 22nd. The Business of SSI is coming up Thursday August 4th. From the EventBrite about the […]

The post Special Topic IIW 1/2 Day Virtual Events – UX July 22nd and Business Aug 4th appeared first on Identity Woman.

Monday, 28. June 2021

DustyCloud Brainstorms

Hello, I'm Chris Lemmer-Webber, and I'm nonbinary trans-femme

I recently came out as nonbinary trans-femme. That's a picture of me on the left, with my spouse Morgan Lemmer-Webber on the right. In a sense, not much has changed, and so much has changed. I've dropped the "-topher" from my name, and given the common tendency to apply gender …

I recently came out as nonbinary trans-femme. That's a picture of me on the left, with my spouse Morgan Lemmer-Webber on the right.

In a sense, not much has changed, and so much has changed. I've dropped the "-topher" from my name, and given the common tendency to apply gender to pronouns in English, please either use nonbinary pronouns or feminine pronouns to apply to me. Other changes are happening as I wander through this space, from appearance to other things. (Probably the biggest change is finally achieving something resembling self-acceptance, however.)

If you want to know more, Morgan and I did a podcast episode which explains more from my present standing, and also explains Morgan's experiences with being demisexual, which not many people know about! (Morgan has been incredible through this whole process, by the way.)

But things may change further. Maybe a year from now those changes may be even more drastic, or maybe not. We'll see. I am wandering, and I don't know where I will land, but it won't be back to where I was.

At any rate, I've spent much of my life not being able to stand myself for how I look and feel. For most of my life, I have not been able to look at myself in a mirror for more than a second or two due to the revulsion I felt at the person I saw staring back at me. The last few weeks have been a shift change for me in that regard... it's a very new experience to feel so happy with myself.

I'm only at the beginning of this journey. I'd appreciate your support... people have been incredibly kind to me by and large so far but like everyone who goes through a process like this, it's very hard in those experiences where people aren't. Thank you to everyone who has been there for me so far.


Phil Windley's Technometria

Building an SSI Ecosystem: Health Passes and the Design of an Ecosystem of Ecosystems

Summary: Ever since the Covid pandemic started in 2020, various groups have seen verifiable credentials as a means for providing a secure, privacy-respecting system for health and travel data sharing. This post explores the ecosystem of ecosystems that is emerging as hundreds of organizations around the world rise to the challenge of implementing a globally interoperable system that also resp

Summary: Ever since the Covid pandemic started in 2020, various groups have seen verifiable credentials as a means for providing a secure, privacy-respecting system for health and travel data sharing. This post explores the ecosystem of ecosystems that is emerging as hundreds of organizations around the world rise to the challenge of implementing a globally interoperable system that also respects individual choice and privacy.

In The Politics of Vaccination Passports, I wrote about the controversy surrounding proposals to require people to show proof of vaccination to travel–specifically fly. The worry is that once we get used to presenting a heath credential to travel, for example, it could quickly spread. Presenting an ID of some kind could become the default–with bars, restaurants, churches, stores, and every other public place saying "papers, please!" before allowing entry.

My personal take is that while I'd rather avoid that scenario, it is likely inevitable. And if that's true, I'd love to have it designed in a way that respects individual choice and personal privacy as much as possible. This is a tall order because getting the tech right is only the first step on a long road. The air travel industry is global, gargantuan, and incredibly decentralized. Building an interoperable system of travel passes requires not just an ecosystem, but an ecosystem of ecosystems.

Suppose Alice lives in Naruba and has been vaccinated against Covid. The local hospital where she was vaccinated has issued a credential to Alice with the data about her doses, the dates, vaccine batch numbers, and so on. Alice is traveling on business to the Emirate of Kunami which requires proof of vaccination. To fly, Alice must present a health pass to the gate agent who works for Kunami Air in the Soji airport. How does Kunami Air know they can trust a credential issued by a hospital they've never heard located in another country?

Building a global, interoperable health and travel pass network requires both technology and governance. This sort of situation isn't new. In fact, if we replaced "proof of vaccination" with "airline ticket" in the preceding scenario, we wouldn't think anything of it. Different companies from different countries have been figuring out how to interoperate and trust the results for decades, maybe centuries.

But doing that digitally, and quickly, is a big job with lots of moving parts. Below I take a look at a few of those parts and how they come together to solve the problem.

Good Health Pass Collaborative

The Good Health Pass Collaborative (GHPC) is an "open, inclusive, cross-sector initiative" of dozens of companies from the travel, health, and technology industries that is defining the blueprint for health and travel passes that are privacy-protecting, user-controlled, equitable, globally interoperable, and universally-accepted by international travel.

GHPC is very specific about what a "pass" is:

A credential to which data minimization and anti-correlation have been applied and any relevant travel data has been added so it includes only what a verifier needs to make a trust decision in a specific context (such as boarding a plane).

A credential could have lots of health and travel data. The pass contains just the data needed for a specific context.

GHPC published a set of ten principles in February to guide their work. These principles lay out a path that is consistent with self-sovereignty, respects individual autonomy, and promotes good privacy practices.

In June, GHPC published a blueprint that provides recommendations for building a good health pass. The challenge of meeting the principles, while exchanging health data across different industry sectors can be overwhelming. The blueprint provides specific guidance on how to meet this challenge without sacrificing the principles that make a health pass "good." The recommendations include designs for trust registries and frameworks, data and protocol standards, and other components for the global interoperability of COVID certificate ecosystems.

These efforts say what a good health pass is and give guidance for creating a credential ecosystem, but they don't create any such ecosystems. That's a job for other organizations.

The Global COVID Certificate Network

The GHCP Blueprint provides recommendations about trust registries and frameworks, data and protocol standards, and other requirements to create global, interoperability ecosystems for COVID certificates. The Linux Foundation's Global COVID Certificate Network (GCCN) "operationalizes and adapts the GHCP Blueprint recommendations for governments and industry alliances who are working to reopen borders." You can think of GCCN as an instantiation of the GHPC Blueprint.

Going back to our traveler, Alice, we can imagine that Naruba and Kunami both have national health systems that can follow the blueprint from GHCP. When Alice uses her health pass inside Naruba, it is reasonable to expect that Narubian businesses will know what a real Narubian health pass looks like and whether to trust it or not. To make this possible, the Narubian health ministry would determine what data a legitimate pass contains (e.g. its schema) and what forms it takes such as the paper design and digital format. The ministry could also determine who in Naruba can issue these health passes and even set up a registry so others in Naruba can find out as well.

This kind of one-size-fits-all, single-source solution can solve the local problem, but when Alice is interacting with people and organizations outside Naruba, the problem is much harder:

Naruba and Kunami may have adopted different technologies and schema for the credential representing the health pass. Random organizations (and even people) in Kunami need to be able to establish the fidelity of the credential. Specifically, they want to know that it was issued to the person presenting it, hasn't been tampered with, and hasn't been revoked. In addition, these same entities need to be able to establish the provenance of the credential, specifically that it was issued by a legitimate organization who is authorized to attest to the holder's vaccination status.

This is where GCCN comes in. GCCN has three components:

a trust registry network a certificate implementation toolkit a set of recommended vendors

The trust registry network and its associated protocol not only helps Naruba and Kunami each establish their own registries of authorized health pass credential issuers, but also enables a directory of registries, so an organization in Kunami can reliably find the registry in Naruba and discover if the issuer of Alice's credential is in it.

The toolkit provides several important components for a working ecosystem:

a template for a governance framework that governments and industry alliances can use to make their own policies. schema definitions and minimum data sets for the credentials. technical specifications for the software components needed to issue, hold, and verify credentials. implementation guides and open source reference implementations. guidance for creating the governance framework and technical implementation.

The vendor network provides a commercial ecosystem to which governments and industry associations can turn for support. The vendor network provides a set of organizations who have competence in building credential ecosystems. Over 25 separate companies and organizations support GCCN.

With all this, GCCN doesn't actually build the ecosystems. That falls to organizations who use GCCN to instantiate the framework provided by GHPC.

Building the Ecosystem

One of those organizations is Cardea. Cardea is a fully open-source and field-tested verifiable digital credential that meets the major requirements of the Good Health Pass Blueprint. Cardea was developed by Indicio and is now a community-led project at Linux Foundation Public Health (LFPH).

Cardea was trialed on the island of Aruba by SITA, one of the participating companies in the Cardea initiative. SITA is a good partner for this since they're responsible for a majority of airline industry IT systems. In the pilot, travelers could prove their Covid test status at restaurants and other tourist locations around the island. The combination of a good trust framework, and self-sovereign-identity-based credential technology allowed businesses to trust tourists' health information while preserving their privacy.

Another example of a working health pass credential ecosystem is the IATA Travel Pass. Travel Pass has been developed by the International Air Transport Association (IATA) and leverages verifiable credential technology from Evernym. The IATA Travel Pass is conducting initial pilots with over 50 airlines and the International Airlines Group (IAG).

A third example is Medcreds from ProofMarket. Medcreds uses the Trinsic API to provide service. MedCreds partnered with the Georgia Academy of General Dentistry to provide free COVID-19 test credentials to their providers and patients. MedCreds allows dentists to reduce the risk of spreading and contracting COVID-19 by knowing with greater certainty the status of a patient's most recent test.

Cardea, IATA Travel Pass, and Medcreds have tools for hospitals, labs, and other organizations who issue passes and for the airlines and other venues who verify them. All also have wallet apps for credential holders to use in receiving credentials as well as presenting the proofs that constitute the actual travel pass from those credentials. In addition, all three initiatives include governance frameworks that service their respective ecosystem.

Because all three are compliant with the GHPC's Blueprint and GCCN's architecture and guidance, the ecosystems they each create are part of the global ecosystem of ecosystems for health and travel passes. As more organizations, countries, and technical teams join in this, the global ecosystem of ecosystems will constitute the largest ever verifiable credential use case to date. The planning, design, and implementation show the level of effort necessary to create large credential ecosystems and provide a path for others to follow.

As I said in The Politics of Vaccination Passports, I'm persuaded that organizations like the Good Health Pass Collaborative and Global Covid Credential Network aren't the bad guys. They're just folks who see the inevitability of health and travel passes and are determined to see that it's done right, in ways that respect individual choice and personal privacy as much as possible.

Tags: verifiable+credentials identity ssi use+cases healthcare


Werdmüller on Medium

Storytelling for Ma

Coping with my mother’s death by telling her a story. Continue reading on Medium »

Coping with my mother’s death by telling her a story.

Continue reading on Medium »


Hyperonomy Digital Identity Lab

Trusted Digital Web: 8-Layer Architecture Reference Model (TDW-ARM)

Github: https://github.com/mwherman2000/TrustedDigitalWeb After about 2.5 years, I finally have an ARM that I like and a code base that is starting to show some promise… 8-Layer Architecture Reference Model (TDW-ARM) Click the model to enlarge it. Trusted Digital Assistant (TDA) … Continue reading →

Github: https://github.com/mwherman2000/TrustedDigitalWeb

After about 2.5 years, I finally have an ARM that I like and a code base that is starting to show some promise…

8-Layer Architecture Reference Model (TDW-ARM)

Click the model to enlarge it.

Trusted Digital Assistant (TDA) Prototype

Github: https://github.com/mwherman2000/TrustedDigitalWeb

Microsoft “Trinity” Graph Engine

Web site: https://www.graphengine.io/

Github: https://github.com/microsoft/GraphEngine

Verifiable Credential Notarization: User Scenarios 0.25 Verifiable Notarization Protocol (VCNP) 0.25 TDW Agents, Wallets, and VDR: ARM

Damien Bod

Sign-in using multiple clients or tenants in ASP.NET Core and Azure AD

The article shows how an ASP.NET Core application could implement a sign in and a sign out with two different Azure App registrations which could also be implemented using separate identity providers (tenants). The user of the application can decide to authenticate against either one of the Azure AD clients. The clients can also be […]

The article shows how an ASP.NET Core application could implement a sign in and a sign out with two different Azure App registrations which could also be implemented using separate identity providers (tenants). The user of the application can decide to authenticate against either one of the Azure AD clients. The clients can also be deployed on separate Azure Active directories. Separate authentication schemes are used for both of the clients. Each client requires a scheme for the Open ID Connect sign in and the cookie session. The Azure AD client authentication is implemented using Microsoft.Identity.Web.

Code: https://github.com/damienbod/AspNetCore6Experiments

The clients are setup to use a non default Open ID Connect scheme and also a non default cookie scheme. After a successful authentication, the OnTokenValidated event is used to sign into the default cookie scheme using the claims principal returned from the Azure AD client. “t1” is used for the Open ID Connect scheme and “cookiet1” is used for the second scheme. No default schemes are defined. The second Azure App Registration client configuration is setup in the same way.

services.AddAuthentication() .AddMicrosoftIdentityWebApp( Configuration.GetSection("AzureAdT1"), "t1", "cookiet1"); services.Configure<OpenIdConnectOptions>("t1", options => { var existingOnTokenValidatedHandler = options.Events.OnTokenValidated; options.Events.OnTokenValidated = async context => { await existingOnTokenValidatedHandler(context); await context.HttpContext.SignInAsync( CookieAuthenticationDefaults .AuthenticationScheme, context.Principal); }; }); services.AddAuthentication() .AddMicrosoftIdentityWebApp( Configuration.GetSection("AzureAdT2"), "t2", "cookiet2"); services.Configure<OpenIdConnectOptions>("t2", options => { var existingOnTokenValidatedHandler = options.Events.OnTokenValidated; options.Events.OnTokenValidated = async context => { await existingOnTokenValidatedHandler(context); await context.HttpContext.SignInAsync( CookieAuthenticationDefaults .AuthenticationScheme, context.Principal); }; });

The AddAuthorization is used in a standard way and no default policy is defined. We would like the user to have the possibility to choose against what tenant and client to authenticate.

services.AddAuthorization(); services.AddRazorPages() .AddMvcOptions(options => { }) .AddMicrosoftIdentityUI();

A third default scheme is added to keep the session after a successful authentication using the client schemes which authenticated. The identity is signed into this scheme after a successfully Azure AD authentication. The SignInAsync method is used for this in the OnTokenValidated event.

services.AddAuthentication(CookieAuthenticationDefaults.AuthenticationScheme) .AddCookie();

The Configure method is setup in a standard way.

public void Configure(IApplicationBuilder app) { app.UseHttpsRedirection(); app.UseStaticFiles(); app.UseRouting(); app.UseAuthentication(); app.UseAuthorization(); app.UseEndpoints(endpoints => { endpoints.MapRazorPages(); endpoints.MapControllers(); }); }

The sign in and the sign out needs custom implementations. The SignInT1 method is used to authenticate using the first client and the SignInT2 is used for the second. This can be called from the Razor page view. The CustomSignOut is used to sign out the correct schemes and redirect to the Azure AD endsession endpoint. The CustomSignOut method uses the clientId of the Azure AD configuration to sign out the correct session. This value can be read using the aud claim.

using System.Threading.Tasks; using Microsoft.AspNetCore.Authentication; using Microsoft.AspNetCore.Authentication.Cookies; using Microsoft.AspNetCore.Authorization; using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.Configuration; namespace AspNetCoreRazorMultiClients { [AllowAnonymous] [Route("[controller]")] public class CustomAccountController : Controller { private readonly IConfiguration _configuration; public CustomAccountController(IConfiguration configuration) { _configuration = configuration; } [HttpGet("SignInT1")] public IActionResult SignInT1([FromQuery] string redirectUri) { var scheme = "t1"; string redirect; if (!string.IsNullOrEmpty(redirectUri) && Url.IsLocalUrl(redirectUri)) { redirect = redirectUri; } else { redirect = Url.Content("~/")!; } return Challenge(new AuthenticationProperties { RedirectUri = redirect }, scheme); } [HttpGet("SignInT2")] public IActionResult SignInT2([FromQuery] string redirectUri) { var scheme = "t2"; string redirect; if (!string.IsNullOrEmpty(redirectUri) && Url.IsLocalUrl(redirectUri)) { redirect = redirectUri; } else { redirect = Url.Content("~/")!; } return Challenge(new AuthenticationProperties { RedirectUri = redirect }, scheme); } [HttpGet("CustomSignOut")] public async Task<IActionResult> CustomSignOut() { var aud = HttpContext.User.FindFirst("aud"); if (aud.Value == _configuration["AzureAdT1:ClientId"]) { await HttpContext.SignOutAsync(CookieAuthenticationDefaults.AuthenticationScheme); await HttpContext.SignOutAsync("cookiet1"); var authSignOut = new AuthenticationProperties { RedirectUri = "https://localhost:44348/SignoutCallbackOidc" }; return SignOut(authSignOut, "t1"); } else { await HttpContext.SignOutAsync(CookieAuthenticationDefaults.AuthenticationScheme); await HttpContext.SignOutAsync("cookiet2"); var authSignOut = new AuthenticationProperties { RedirectUri = "https://localhost:44348/SignoutCallbackOidc" }; return SignOut(authSignOut, "t2"); } } } }

The _LoginPartial.cshtml Razor view can use the CustomAccount controller method to sign in or sign out. The available clients can be selected in a drop down control.

<ul class="navbar-nav"> @if (User.Identity.IsAuthenticated) { <li class="nav-item"> <span class="navbar-text text-dark">Hello @User.Identity.Name!</span> </li> <li class="nav-item"> <a class="nav-link text-dark" asp-controller="CustomAccount" asp-action="CustomSignOut">Sign out</a> </li> } else { <li> <div class="main-menu"> <div class="dropdown"> <button class="btn btn-primary dropdown-toggle" type="button" id="dropdownLangButton" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> sign-in </button> <div class="dropdown-menu" aria-labelledby="dropdownLangButton"> <a class="dropdown-item" asp-controller="CustomAccount" asp-action="SignInT1" >t1</a> <a class="dropdown-item" asp-controller="CustomAccount" asp-action="SignInT2">t2</a> </div> </div> </div> </li> } </ul>

The app.settings have the Azure AD settings for each client as required.

{ "AzureAdT1": { "Instance": "https://login.microsoftonline.com/", "Domain": "damienbodhotmail.onmicrosoft.com", "TenantId": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "ClientId": "46d2f651-813a-4b5c-8a43-63abcb4f692c", "CallbackPath": "/signin-oidc/t1", "SignedOutCallbackPath ": "/SignoutCallbackOidc" // "ClientSecret": "add secret to the user secrets" }, "AzureAdT2": { "Instance": "https://login.microsoftonline.com/", "Domain": "damienbodhotmail.onmicrosoft.com", "TenantId": "7ff95b15-dc21-4ba6-bc92-824856578fc1", "ClientId": "8e2b45c2-cad0-43c3-8af2-b32b73de30e4", "CallbackPath": "/signin-oidc/t2", "SignedOutCallbackPath ": "/SignoutCallbackOidc" // "ClientSecret": "add secret to the user secrets" },

When the application is started, the user can login using any client as required.

This works really good, if you don’t know which tenant is your default scheme. If you always use a default scheme with one tenant default, then you can use the multiple-authentication-schemes example like defined in the Microsoft.Identity.Web docs.

Links:

https://github.com/AzureAD/microsoft-identity-web/wiki/multiple-authentication-schemes

https://github.com/AzureAD/microsoft-identity-web/wiki/customization#openidconnectoptions

https://github.com/AzureAD/microsoft-identity-web

https://docs.microsoft.com/en-us/aspnet/core/security/authentication

Saturday, 26. June 2021

Timothy Ruff

Verifiable Credentials Aren’t Credentials. And They’re Not Verifiable In the Way You Might Think.

Last summer I published a 3-part series about the similarities between verifiable credentials (VCs) and physical shipping containers: Part 1: Verifiable Credentials Aren’t Credentials. They’re Containers. Part 2: Like Shipping Containers, Verifiable Credentials Will Economically Transform the World Part 3: How Verifiable Credentials Bridge Trust Domains The similarities have only bec

Last summer I published a 3-part series about the similarities between verifiable credentials (VCs) and physical shipping containers:

Part 1: Verifiable Credentials Aren’t Credentials. They’re Containers.

Part 2: Like Shipping Containers, Verifiable Credentials Will Economically Transform the World

Part 3: How Verifiable Credentials Bridge Trust Domains

The similarities have only become more apparent, and important, now that VCs are rapidly growing in global adoption. So it’s time for an update, including some history behind the VC name and why “verifiable” is also problematic.

It can be found at my new blog location: https://credentialmaster.com/verifiable-credentials-arent-credentials-theyre-containers/


The Verifiable Credentials (VC) Stack

SSI and verifiable credentials (VCs) have spawned hundreds of startups and significant initiatives within Microsoft, IBM, Workday, Samsung, Deutsche Telekom, the entire EU, the U.S. Government, the global credit union industry, 40+ airlines, hundreds of healthcare organizations, GLEIF (international financial regulator), GS1 (global standards body for QR and bar codes), and more. How do the

SSI and verifiable credentials (VCs) have spawned hundreds of startups and significant initiatives within Microsoft, IBM, Workday, Samsung, Deutsche Telekom, the entire EU, the U.S. Government, the global credit union industry, 40+ airlines, hundreds of healthcare organizations, GLEIF (international financial regulator), GS1 (global standards body for QR and bar codes), and more.

How do the various types of products and services fit together?

Which are complementary, and which are competitive?

Should the services from provider A work with those from provider B?

Will you still need provider B if you have provider A?

Without naming specific providers, “The VC Stack” is Credential Master’s attempt to develop a framework for answering these questions at the highest, simplest, non-technical level possible. It is well over a year in the making, with input from many of the world’s top experts in SSI.

The VC Stack considers VCs as the foundational building block for SSI, rather than identity, blockchain, or anything else. It organizes VCs into four essential functions: Applications, Management, Processing, and Storage.

This separation clarifies product functions and vendor roles, expands administrative capabilities, reduces vendor lock-in, lessens the impact of changes in VC standards and technology, and enables service providers to focus on what they do best.

Learn more from my recent detailed blog: https://credentialmaster.com/the-vc-lifecycle/. We invite any and all feedback to make it better.

Friday, 25. June 2021

Werdmüller on Medium

The open banking elephant

The American banking system sucks. It’s time for open banking. Continue reading on Medium »

The American banking system sucks. It’s time for open banking.

Continue reading on Medium »

Thursday, 24. June 2021

Doc Searls Weblog

A storage crisis

The best new phones come with the ability to shoot 108 megapixel photos, record 4K video with stereo sound, and pack the results into a terabyte of onboard storage. But what do you do when that storage fills up? If you want to keep those files, you’ll need to offload them somewhere. Since your computer […]

The best new phones come with the ability to shoot 108 megapixel photos, record 4K video with stereo sound, and pack the results into a terabyte of onboard storage. But what do you do when that storage fills up?

If you want to keep those files, you’ll need to offload them somewhere. Since your computer probably doesn’t have more than 2Tb of storage, you’ll need an external drive. Or two. Or three. Or more. Over time, a lot more.

Welcome to my world.

Gathered here for a portrait in a corner of my desk in Manhattan are 22 hard drives, plus three SD cards that each exceed the capacities of the drives they’re laying on. And then there’s the 2Tb one in the laptop I’m using now. That one has 357.33Gb available. Most of the others you see are also full, dead, or both. Five have FireWire connections, which my current laptop doesn’t comprehend at all. I also have a similar collection of drives in Santa Barbara.

Photos occupy most of the data I’ve stored on all those drives. Currently my photo archives are spread between two portable drives and my laptop, and total about 7Tb. I also have a 5Tb portable drive for videos, which is back in Santa Barbara awaiting dubs off tapes. The portable photo drives are among those in the picture above. Earlier today, my laptop gave me this news about the main one, called Black 4Tb WD Photo Drive:

That’s why I’m transferring its contents over to the 10Tb drive called Elements, on the far left. A progress report:

About 5Tb of Elements is occupied by Apple Time Machine backups. After the transfer is done, there won’t be room for more backups. So my project now is figuring out what to do next.

I could get some Network Attached Storage (NAS). I already have used 2012-vintage 18Tb QNAP one in Santa Barbara that I’ve never been able to make work.

So I am tempted now to put it all in a cloud.

I hadn’t bothered with that before, because upstream speeds have been highly sphinctered by ISPs for decades. Here in New York, where our ISP is Spectrum, our speeds have long run 100-400 Mbps down, but only 10 Mbps up. However…. I just checked again with Speedtest.net, and got this:

And that’s over wi-fi.

Now I’m encouraged. But before I commit to a supplier, I’d like to hear what others recommend. Currently I’m considering Backblaze, which is top rated here. The cost i $6/month, or less for unlimited sums of data. But I’m open to whatever.

[Later…] Hmm. At that last link it says this:

What We Don’t Like:

Something I should mention is that some users have had bad experiences with Backblaze because of a not-so-apparent feature that maybe should be a lot more obvious: Backblaze doesn’t function as a permanent archive of all of your data, but instead as a mirror.

In other words, if you delete files on your computer, or the drive fails and you’re connected to Backblaze’s website, Backblaze will see that those files are gone and will remove them from your online account, too.

Granted, signing up for the forever version history option would eliminate any issues with this, but it still poses a problem for anyone using one of the limited version history options.

Alas, the forever thing is complicated.

To be clear, I want more than a mirroring of what I have on my laptop and external drives. I want to replace those external drives with cloud storage. Is that possible? Not clear.

Alas, for all of us, this problem remains.

Oh, and Spectrum now only measures under 10Mbps upstream. So forget the cloud.


Redux 002: Listen Up

This is a 1999 post on the (pre-blog) website that introduced my handful of readers to The Cluetrain Manifesto, which had just gone up on the Web, and instantly got huge without my help. It was also a dry run for a chapter in the book by the same name, which came out in January, 2000. As best I can recall, […]

This is a 1999 post on the (pre-blog) website that introduced my handful of readers to The Cluetrain Manifesto, which had just gone up on the Web, and instantly got huge without my help. It was also a dry run for a chapter in the book by the same name, which came out in January, 2000. As best I can recall, I wrote most of it a year earlier, and updated it when Cluetrain was finally published.

Listen Up

By Doc Searls
April 16, 1999 

“All I know is that first you’ve got to get mad. You’ve got to say, I’m a human being, goddammit! My life has value! So I want you to get up now. I want all of you to get up out of your chairs. I want you to get up right now and go to the window, open it, and stick your head out, and yell, ‘I’m as mad as hell, and I’m not going to take this anymore!'”
— Howard Beale, in Network, by Paddy Chayevsky

Bob Davis is the CEO of Lycos, Inc., whose growing portfolio of companies (excuse me, portals) now includes LycosHotbotWhoWhere and Tripod. I’m sure Bob is a great guy. And I’m sure Lycos is a great company. A lot of people seem to like them both. And you have to admire both his ambition and his success. To witness both, read his interview with PC Week, where he predicts that the Lycos Network (the sum of all its portals) will overtake Yahoo as “#1 on the Web.”

Lycos will win, Davis says, because “We have a collection of quality properties that are segmented into best-of-breed categories, and our reach has been catapulting.”

I can speak for Hotbot, which is still my first-choice search engine; but by a shrinking margin. I often test search engines by looking for strings of text buried deep in long documents on my own site. Hotbot always won in the past. But since Lycos bought it, Hotbot has become more of a portal and less of a search tool. Its page is now a baffling mass of ads and links. And its searches find less.

In today’s test, Infoseek won. Last week, Excite won. Both found pages that Hotbot seems to have forgotten.

Why? Bob Davis gives us a good answer.

“We’re a media company,” he says. “We make our money by delivering an audience that people want to pay for.”

Note the two different species here: audience and people. And look at their qualities. One is “delivered.” The other pays. In other words, one is cargo and the other is money.

Well, I don’t care if Lycos’ stock goes to the moon and splits three times along the way. The only #1 on the Web is the same as the only #1 on the phone: the people who use it. And the time will come when people will look at portals not as sources of “satisfying experiences” (another of Davis’ lines) but as useless intermediaries between supply and demand.

 

Words of Walt

You there, impotent, loose in the knees,
open your scarfed chops till I blow grit within you.
Spread your palms and lift the flaps of your pockets.
I am not to be denied. I compel.
It is time to explain myself. Let us stand up.
I know I am solid and sound.
To me the converging objects of the universe perpetually flow.

I know that I am august,
I do not trouble my spirit to vindicate itself
or be understood.
I see that the elementary laws never apologize.

.
— Walt Whitman, from Song of Myself

 

“Media company” guys like Davis are still in a seller’s market for wisdom that was BS even when only the TV guys spoke it — back when it literally required the movie “Network.” That market will dry up. Why? Because we’ve been mad as hell for about hundred years, and now we don’t have to take it anymore.

Three reasons.

Humanity. This is what Walt Whitman reminded us about more than a hundred years ago. We are not impotent. Media companies may call us seats and eyeballs and targets, but that’s their problem. They don’t get who we are or what we can — and will — do. And the funny thing is, they don’t get that what makes us powerful is what they think makes them powerful: the Internet. It gives us choices. Millions of them. We don’t have to settle for “channels” any more. Or “portals” that offer views of the sky through their own little windows. Or “sticky” sites that are the moral equivalent of flypaper. Demand. There never was a demand for messages, and now it shows, big time. Because the Internet is a meteor that is smacking the world of business with more force than the rock that offed the dinosaurs, and it is pushing out a tsunami of demand like nothing supply has ever seen. Businesses that welcome the swell are in for some fun surfing. Businesses that don’t are going to drown in it. Obsolescence. Even the media guys are tired of their own B.S. and are finally in the market for clues.

Alvin Toffler had it right in The Third Wave. Industry (The Second Wave) “violently split apart two aspects of our lives that had always been one… production and consumption… In so doing, it drove a giant invisible wedge into our economy, our psyches … it ripped apart the underlying unity of society, creating a way of life filled with economic tension.” Today all of us play producer roles in our professions and consumer roles in our everyday lives. This chart shows the difference (and tension) between these radically different points of view — both of which all of us hold:

Producer view Consumer view Metaphor Business is shipping (“loading the channel,” “moving products,” “delivering messages”) Business is shopping (“browsing,” “looking,” “bargaining,” “buying”) Orientation Business is about moving goods from one to many (producers to consumers) Business is about buying and selling, one to one Markets Markets are shooting ranges: consumers are “targets” Markets are markets: places to shop, buy stuff and talk to people Relationships Primary relationshiphs are with customers, which are more often distributors & retailers rather than consumers Primary relationships are with vendors, and with other customers

 

These are all just clues, which are easily deniable facts. Hence a line once spoken of Apple: “the clue train stopped there four times a day for ten years and they never took delivery.” But Apple was just an obvious offender. All of marketing itself remains clueless so long as it continues to treat customers as “eyeballs,” “targets,” “seats” and “consumers.”

For the past several months, I have been working with Rick Levine, David Weinberger and Chris Locke on a new railroad for clues: a ClueTrain.

Our goal is to burn down Marketing As Usual. Here is the logic behind the ambition:

Markets are conversations
Conversations are fire
Marketing is arson

The result is here — in what The Wall Street Journal calls “presumptuous, arrogant, and absolutely brilliant.”

Take a ride. If you like it, sign up. Feel free to set fires with it, add a few of your own, or flame the ones you don’t agree with. What matters is the conversation. We want everybody talking about this stuff. If they do, MAU is toast.

Here is my own short form of the Manifesto (inspired by Martin Luther, the long version has 95 Theses). Feel free to commit arson with (or to) any of these points as well.

Ten facts about highly effective markets: Markets are conversations.
None of the other metaphors for markets — bulls, bears, battlefields, arenas, streets or invisible hands — does full justice to the social nature of markets. Real market conversations are social. They happen between human beings. Not between senders and receivers, shooters and targets, advertisers and demographics. The first markets were markets.
They were real places that thrived at the crossroads of cultures. They didn’t need a market model, because they were the model market. More than religion, war or family, markets were real places where communities came together. They weren’t just where sellers did business with buyers. They were the place where everybody got together to hang out, talk, tell stories and learn interesting stuff about each other and the larger world. ‘ Markets are more about demand than supply.
The term “market” comes from the latin mercere, which means “to buy.” Even a modern market is called a “shopping center” rather than a “selling center.” Bottom line: every market has more buyers than sellers. And the buyers have the money. Human voices trump robotic ones.
Real voices are honest, open, natural, uncontrived. Every identity that speaks has a voice. We know each other by how we sound. That goes for companies and markets as well as people. When a voice is full of shit, we all know it — whether the voice tells us “your call is important to us” or that a Buick is better than a Mercedes. The real market leaders are people whose minds and hands are worn by the work they do.
And it has been that way ever since our ancestors’ authority was expressed by surnames that labeled their occupations — names like Hunter, Weaver, Fisher and Smith. In modern parlance, the most knowledge and the best expertise is found at the “point of practice:” That’s where most of the work gets done. Markets are made by real people.
Not by surreal abstractions that insult customers by calling them “targets,” “seats,” “audiences,” “demographics” and “eyeballs” — all synonyms for consumers, which Jerry Michalski of Sociate calls “brainless gullets who live only to gulp products and expel cash.” Business is not a conveyor belt that runs from production to consumption.
Our goods are more than “content” that we “package” and “move” by “loading” them into a “channel” and “address” for “delivery.” The business that matters most is about shopping, not shipping. And the people who run it are the customers and the people who talk to them. Mass markets have the same intelligence as germ populations.
Their virtues are appetite and reproduction. They grow by contagion. Which is why nobody wants to admit belonging to one. There is no demand for messages.
To get what this means, imagine what would happen if mute buttons on remote controls delivered “we don’t want to hear this” messages directly back to advertisers. Most advertising is unaccountable.
Or worse, it’s useless. An old advertising saying goes, “I know half my advertising is wasted. I just don’t know which half.” But even this is a lie. Nearly all advertising is wasted. Even the most accountable form of advertising — the junk mail we euphemistically call “direct marketing” — counts a 3% response rate as a success. No wonder most of us sort our mail over the trash can. Fairfax Cone, who co-founded Foote Cone & Belding many decades ago, said “Advertising is what you do when you can’t go see somebody. That’s all it is.” With the Net you can go see somebody. More importantly, they can see you. More importantly than that, you can both talk to each other. And make real markets again.

What becomes of journalism when everybody can write or cast?

Formalized journalism is outnumbered. In the industrialized world (and in much of the world that isn’t), nearly everyone of a double-digit age has a Net-connected mobile device for sharing words they write and scenes they shoot. While this doesn’t obsolesce professional journalists, it marginalizes and re-contextualizes them. Worse, it exposes the blindness within their formalities. Dave […]

Formalized journalism is outnumbered.

In the industrialized world (and in much of the world that isn’t), nearly everyone of a double-digit age has a Net-connected mobile device for sharing words they write and scenes they shoot.

While this doesn’t obsolesce professional journalists, it marginalizes and re-contextualizes them. Worse, it exposes the blindness within their formalities. Dave Winer lays this out clearly in They can’t see what they can’t see. An excerpt:

Journalism, academia, government and the corporate world all hire from the same talent pool.

They go to the same universities, get their news from the same sources. Corporate people take government jobs, then go back to the corporations. The people move fluidly in and out of each bucket.

So you get the same story, the reality they believe in, developed over centuries, that is radically different from the reality most other people experience. The story recited daily at CNN, MSNBC, The New Yorker, NYT. The world changes, again and again, and the story they tell is how angry this makes them, and how everyone must snap back.

New technologies can make change possible, like the one we’re using now.

We would never have Trump if it weren’t for Twitter.

Marshall McLuhan, dead since 1980, had something to say about that:

People don’t want to know the cause of anything. They do not want to know why radio caused Hitler and Gandhi alike. They do not want to know that print caused anything whatever.

Also,

All media work us over completely. They are so pervasive… that they leave no part of us untouched unaffected, unaltered… Any understanding of social and cultural change is impossible without a knowledge of the way media work as environments.

What could be more worked-over than our new nature as digital beings? Consider where and what you are reading right now, on your desk, your lap or in your hand. This was barely imaginable when the word “journalism” came into use around 1830.

What are we now? While remaining no less embodied than we ever were, we are incorporeal inhabitants of a digital world, absent of distance and gravity, where the only preposition that fully applies is with. Unless we aren’t. Ephemerality runs deep, and the Web is a whiteboard.

I don’t have an answer to the question in my headline. I tried one out three years ago, in my first, last and only TEDx talk, The story isn’t the whole story. To summarize, we gotta start local.

Because it is essential to be real with each other in the physical world, especially when fully relevant news actually happens. Conspiracy theories and other forms of bullshit will always be everywhere, but it’s harder for them to apply or survive when your building is on fire or your neighborhood is flooding.

Outside of that, I’m starting to think we need a new discipline: one that doesn’t start with the word journal. And decades may pass before we have it.

Wednesday, 23. June 2021

Mike Jones: self-issued

Second Version of FIDO2 Client to Authenticator Protocol (CTAP) Now a Standard

The FIDO Alliance has completed the CTAP 2.1 Specification. This follows the publication of the closely-related second version of the W3C Web Authentication (WebAuthn) specification. Today’s FIDO Alliance announcement describes the enhancements in the second version as follows: Enhancements to FIDO standards to accelerate passwordless in the enterprise The FIDO Alliance has announced enhancements t

The FIDO Alliance has completed the CTAP 2.1 Specification. This follows the publication of the closely-related second version of the W3C Web Authentication (WebAuthn) specification.

Today’s FIDO Alliance announcement describes the enhancements in the second version as follows:

Enhancements to FIDO standards to accelerate passwordless in the enterprise

The FIDO Alliance has announced enhancements to its FIDO2 specifications, which include several new features that will be helpful for passwordless enterprise deployments and other complex security applications. Both FIDO2 specifications were recently updated by their governing bodies – with the World Wide Web Consortium (W3C) approving WebAuthn Level 2 and FIDO doing the same for CTAP 2.1.

Key to these enhancements is enterprise attestation, which provides enterprise IT with improved management of FIDO authenticators used by employees. Enterprise attestation enables better binding of an authenticator to an account, assists with usage tracking and other management functions including credential and pin management, and biometric enrollment required in the enterprise.

Other updates include support for cross-origin iFrames and Apple attestation, as well as improvements to resident credentials. More details on these and other FIDO specification enhancements are available here.


@_Nat Zone

Identiverse: 7 Principles of Digital Being

日本時間 7月1日午前4時 (Wednesday,… The post Identiverse: 7 Principles of Digital Being first appeared on @_Nat Zone.

日本時間 7月1日午前4時 (Wednesday, June 30 1:00pm – 1:30pm MDT)

2020年に、物理的な世界からサイバー世界への移行がかつてないスピードで進みました。COVID-19がその移行を課したのです。5年から10年分の変化が数ヶ月の間に行われたと言う人もいます。そこでは、ステイホーム・ポリシーにより物理的な世界でのコミュニケーション能力が大きく制限され、サイバー・コミュニケーションに大きく依存するようになりました。サイバー空間では、あらゆるコミュニケーションはDigital Being(サイバースペースにおける私たちのデジタル表現)を通じて行われます。しかし、Digital Beingが無形である以上、私たちの「安全な生活」に欠かせない、当たり前だと思っている物理的な存在の多くの特性がそこにはありません。このセッションでは、私たちのデジタルライフを安全で安心なものにするために必要な、Digital Beingの7つの原則を概観します。

The post Identiverse: 7 Principles of Digital Being first appeared on @_Nat Zone.

Tuesday, 22. June 2021

Doc Searls Weblog

Dialog with a robot

So I ordered one of these yesterday… …and got a notice that said, That was to my building in Manhattan: a three-story walk-up in a district that’s post-dangerous but pre-gentrified. In other words, safe; but not beyond worry that someone could walk off with a box on the steps that says Amazon on it. We […]

So I ordered one of these yesterday…

…and got a notice that said,

That was to my building in Manhattan: a three-story walk-up in a district that’s post-dangerous but pre-gentrified. In other words, safe; but not beyond worry that someone could walk off with a box on the steps that says Amazon on it.

We live on the top floor, so it took me a minute to get from my desk to the front door, where there was nothing. So I went to Amazon’s Contact Us page and got this:

Then, after I clicked on the big yellow button, a dialog ensued :

Hi! It’s Amazon’s messaging assistant again.
Amazon eero mesh WiFi system – 2 pack Is this what you need help with?

12:49 PM

Yes, that’s it

12:50 PM

Messaging Assistant | Customer Service

Looks like this item was delivered on Tuesday, Jun 22. Could you confirm what went wrong, so I can help?

12:50 PM

Didn’t get it

12:50 PM

Messaging Assistant | Customer Service

Sorry to hear that. Tell me a little more about what happened.

12:50 PM

Package never arrived

12:50 PM

Messaging Assistant | Customer Service

OK, looks like the delivery status may have been updated too soon. The good news is it should arrive soon. Give us until end of day Wednesday, Jun 23. Really appreciate your patience with this. Is there anything else I can help with?

12:50 PM

No, that’s all

12:50 PM

Messaging Assistant | Customer Service

Thanks for choosing Amazon. I’m here to help whenever you have an issue.

12:50 PM

Messaging Assistant has left. Thanks for contacting Amazon!

This kind of thing has happened before; not just with Amazon, and not just here. In fact, it has happened enough times for me to wonder whether there is a method to the madness.

I mean, is “the delivery status may have been updated too soon” an occurrence that’s common for a reason? Also, might that reason be technical, human, neither or both?

To be fair, this is not a big deal. Progress is how the miraculous becomes mundane, and this is a good example of it the mundane miraculous at work. “No sooner said than shipped” is the new norm. So are conversations with robots.

I’d love it if one could give me answers below, but that feature isn’t working right now. So feel free to tweet yours or something. Thanks!


@_Nat Zone

Identiverse: Where are we with SIOP and DID?

6月23日(水7:30am – 8:2… The post Identiverse: Where are we with SIOP and DID? first appeared on @_Nat Zone.

6月23日(水
7:30am – 8:20am MDT (1:30pm – 2:20pm UTC)

昨年、DIFとOIDFが提携し、DID/VCの世界をOpenID Connectの世界につなげるための作業が進んでます。この過程で、WGは分散型IDシステムが直面する多くの困難な問題に取り組んでいます。

このパネルでは、3人の著名なパネリストとともに、仕様の状況とこれらの「難しい問題」に関する最新の考え方について議論します。

キム・キャメロン氏、『Laws of Identity』の著者、アイデンティティ・ブログ クリスティーナ・ヤスダ氏、マイクロソフト株式会社、アイデンティティ・スタンダード・アーキテクト トバイアス・ルッカー氏、Mattr社 テクニカル・スタンダード・アーキテクト

パネルディスカッションでは、まずキムが高いレベルの問題提起を行います。次に、OpenID AB/Connect WG内のSIOP Special Callの議長であるクリスティーナが、グループの作業の現状を説明します。

そして、ここからがお楽しみです。私はモデレーターとして「難しい質問」に答えていきます。バーチャル会議の技術がどうなるかわかりませんが、可能であれば、Ping Identity社のDavid Waite氏にも質問をしていただくかもしれません。

ご興味のある方は、ぜひお越し下さい。

このイベントへの登録は、こちらのリンクから行えます: https://pheedloop.com/identiverse2021/site/. ディスカウントコード「speaker1492」をご利用いただくと、登録料が10%オフになります。

The post Identiverse: Where are we with SIOP and DID? first appeared on @_Nat Zone.

19th International Conference on Applied Cryptography and Network Security

2021-06-22 に 19th In… The post 19th International Conference on Applied Cryptography and Network Security first appeared on @_Nat Zone.

2021-06-22 に 19th International Conference on Applied Cryptography and Network Security にて基調講演をさせていただきました。タイトルは『Digital Being』。来週6月30日にIdentiverseでやるもののフル版です。

このような機会をいただけたことを感謝いたします。

The post 19th International Conference on Applied Cryptography and Network Security first appeared on @_Nat Zone.

Kyle Den Hartog

Example Design of an Authorization System with Verifiable Credentials and the Tradeoffs

I'll cover an example design which relies upon verifiable credentials for an authorization system and the tradeoffs faced.
Example Design of an Authorization System with Verifiable Credentials and the Tradeoffs Problem summary

In this blog post, I’ll cover an example design that relies upon verifiable credentials where a CFO is delegating access to their assistant to authorize purchases made by the company. The primary focus of this blog post is to highlight the different problems that are likely to occur when going down the path of building an authorization system with verifiable credentials. I’ll be sure to keep things at a higher level so that anyone can understand these tradeoffs, but take you through the details that would be thought through by an architect designing the system. Enjoy, and feel free to leave comments or reach out to me for questions, suggested improvements, or other aspects I didn’t fully consider.

Use case description

In this use case, the issuer is an employer who wishes to provide an employee credential to their CFO. Now their CFO as a part of their job has a lot of capabilities afforded to them as a senior executive. Let’s say this CFO has two abilities. They can sign off on quarterly reports and they can authorize purchases. Now let’s say the employer issues them the following credential:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "https://example.com/employer", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:cfo", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, }, "revocation": { "id": "http://example.gov/revocations/738", "type": "SimpleRevocationList2017" }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "https://example.com/employer/key1" } }

All is good and well, now the CFO can do their job. Now let’s say the CFO has been bombarded with purchase authorization requests recently while trying to prepare the quarterly report and has asked their assistant to assist with the backlog of purchase requests as long as the requests are under $10,000. To do this, the CFO decides to delegate their employee credential to their assistant because the payment authorization system relies upon the jobTitle attribute to determine the capabilities.

So, the CFO delegates the following credential to their assistant and hands over their employee credential too:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "did:example:cfo", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:assistant", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, }, "revocation": { "id": "http://example.gov/revocations/738", "type": "SimpleRevocationList2017" }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:cfo#key1" } }

Side rhetorical consideration around the semantics of this VC as a standalone VC, why is the CFO declaring that their assistant has the jobTitle “Chief Financial Officer” now? This is what I mean when I say using verifiable credentials for authorization systems introduces bad semantics. To skirt around authorization systems we’re bending the semantics of the claims in the credentials in ways that don’t make sense. And that’s ignoring the second and third-order effects that are inherently going to be introduced!

Great, so now the assistant can generate the following delegated verifiable Presentation to authorize payments:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/789", "type": ["VerifiablePresentation"], "verifiableCredential": [ { "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "https://example.com/employer", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:cfo", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, }, "revocation": { "id": "http://example.gov/revocations/738", "type": "SimpleRevocationList2017" }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "https://example.com/employer/key1" } }, { "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/456", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "did:example:cfo", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:assistant", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:cfo#key1" } }], "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "challenge": "c0ae1c8e-c7e7-469f-b252-86e6a0e7387e", "proofPurpose": "authentication", "verificationMethod": "did:example:assistant#key1" } }

Sweet, now the assistant can authorize payments! Let’s review what the steps are for the payment authorization system now with this.

VP Request checks

Check that the verificationMethod is associated with the authentication proofPurpose for the assistant. Check that the verificationMethod for the VP proof is associated with the delegation credential credentialSubject.id Check that the proofValue is valid in the VP proof (the bottom one) based on the associated verificationMethod for the assistant Check that the challenge matches the VP Request

Delegation Credential checks

Check that the verificationMethod is associated with the assertionMethod proofPurpose for the CFO in the delegation credential Check that the verificationMethod in the delegation credential is associated with the issuer property of the delegation credential Check that the proofValue is valid in the delegation credential proof (the middle one) based on the associated verificationMethod for the CFO Check that the credentialSubject.jobTitle has “Chief Financial Officer” in the delegation credential Check that the issuer property in the delegation credential matches the credentialSubject.id in the original credential

Original Credential checks

Check that the verificationMethod is associated with the assertionMethod proofPurpose for the employer in the original credential Check that the verificationMethod in the original credential is associated with the issuer property of the original credential Check that the proofValue is valid in the original credential proof (the top one) based on the associated verificationMethod for the CFO Check that the credentialSubject.jobTitle has “Chief Financial Officer” in the original credential Check that the original credential isn’t revoked

Note to consider: If the credentials in the verifiable presentation are ordered incorrectly that’s going to introduce some problems

But wait how did the authorization system know that the assistant has been attenuated with limited permission to only authorize payments below $10,000? Well, we’ve got a few different ways we could do that, let’s take a look at the ways we can extend this simple delegation pattern.

Extension 1: CFO adds a maximumAuthorizationLimit property to the delegation credential

So in this scenario, the CFO adds maximumAuthorizationLimit property like so:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "did:example:cfo", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:assistant", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, "maximumAuthorizationLimit": 10000 }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:cfo#key1" } }

and the verifiable presentation would now look like this:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "did:example:76e12ec21ebhyu1f712ebc6f1z2", "type": ["VerifiablePresentation"], "verifiableCredential": [ { "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "https://example.com/employer", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:cfo", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, }, "revocation": { "id": "http://example.gov/revocations/738", "type": "SimpleRevocationList2017" }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "https://example.com/employer/key1" } }, { "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "did:example:cfo", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:assistant", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, "maximumAuthorizationLimit": 10000 }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:cfo#key1" } }], "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "challenge": "c0ae1c8e-c7e7-469f-b252-86e6a0e7387e", "proofPurpose": "authentication", "verificationMethod": "did:example:assistant#key1" } }

Awesome, so now we have a verifiable presentation that attenuates the delegated capabilities of making authorization payments and limits the payments to $10000 for the assistant. This new property introduces some new checks though in our business logic.

Check that the credentialSubject.maximumAuthorizationLimit in the delegation credential is below the limit afforded to the credentialSubject.jobTitle Check that the payment hasn’t exceeded the credentialSubject.maximumAuthorizationLimit

But wait, when we delegated the ability for the assistant to authorize payments we also inadvertently allowed the assistant to sign off on the quarterly reports! Oh no! Now, we’ve got a legal liability issue on our hands. We’ll need to build on top of option 1 to make sure that the assistant isn’t signing off on our quarterly reports too and so that we can adhere to the principle of least authority. We’ll cover that in option 2.

Extension 2: limiting the assistant’s abilities to only payment authorization capabilities

Since the CFO has two capabilities, we need to disable the one that would allow the assistant to also sign off on quarterly reports. We’re going to go ahead and do this via the introduction of the canSignQuartlyReports claim.

So now the delegation credential looks like this:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "did:example:cfo", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:assistant", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, "maximumAuthorizationLimit": 10000, "canSignQuartlyReports": false }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:cfo#key1" } }

and the delegated verifiable presentation now looks like this:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "did:example:76e12ec21ebhyu1f712ebc6f1z2", "type": ["VerifiablePresentation"], "verifiableCredential": [ { "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "https://example.com/employer", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:cfo", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, }, "revocation": { "id": "http://example.gov/revocations/738", "type": "SimpleRevocationList2017" }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "https://example.com/employer/key1" } }, { "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "did:example:cfo", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:assistant", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, "maximumAuthorizationLimit": 10000, "canSignQuartlyReports": false }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:cfo#key1" } }], "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "challenge": "c0ae1c8e-c7e7-469f-b252-86e6a0e7387e", "proofPurpose": "authentication", "verificationMethod": "did:example:assistant#key1" } }

Finally, we’ve got a properly attenuated delegation verifiable credential to address our use case! So what’s the issue with this? First off, we’re using only a toy example here where the CFO has only two capabilities. Most businesses aren’t paying CFOs the big bucks to only handle those two things though (If they are let me know, I’m sure I can figure that job out 😊). This means that for every additional capability that we don’t want to be delegated we have to add a permission property. By doing this we’re effectively building a deny list with only one capability removed to enable a single capability. Why not build off of and allow list model instead? (Hint this is the approach ZCAP-LDs take)

So our delegation credential could end up looking something like this:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", { "@vocab": "https://example.com/employer/" } ], "id": "http://example.com/employer/credentials/123", "type": ["VerifiableCredential", "EmployerCredential"], "issuer": "did:example:cfo", "issuanceDate": "2010-01-01T19:73:24Z", "credentialSubject": { "id": "did:example:assistant", "jobTitle": "Chief Financial Officer", "employeeNumber": 123, "maximumAuthorizationLimit": 10000, "canSignQuartlyReports": false, "canDoXCapability": false, "canDoYCapability": false, "canDoZCapability": false, ... }, "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "....", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:cfo#key1" } } Tradeoffs with this design

Furthermore, the semantics of the claims in this new credential is quite odd. We’ve effectively combined a claims token and a permission token into a single verifiable Credential here. By doing this we’re far more likely to be issuing and reissuing the same credential many times as access to the delegatee needs to be updated. This also has second-order effects on the maintainability and adaptability that are best described by the principle of least knowledge.

It also just so happens, these types of deny lists are awesome things to get your hands on as well when you’re performing the reconnaissance phase of a penetration test to break into a system. Now our system design is helping the attackers assuming they get ahold of one of the credentials, but none of the keys. If they get the keys, problems that are better left out of scope for this writeup occur.

Additionally, the fact that we need to perform 16 checks just to achieve our goal means that it’s highly likely that one gets missed and an authorization bypass vulnerability is introduced. As is pointed out in the CCG mailing list though, this is mainly a byproduct of the fact we’re performing delegation rather than the usage of VCs themselves. However, delegation is only complicated when done correctly. After all, when people are sharing their Netflix passwords they too are delegating and that is a simple implementation for the verifier. So the takeaway here is that the more we can reduce the number of checks when building transparent delegation the more resilient and secure the authorization system is likely to be.

Conclusion

In conclusion, I stand by the opinion that utilizing verifiable credentials to build authorization systems is possible, but more work than is necessary. Hopefully, this walk-through helps to understand why I’ve come to this conclusion. However, just because I’ve decided I’m probably not going to use VCs in this way doesn’t mean you shouldn’t. Just like any technology, some tradeoffs occur when deciding how you approach your system design. It may be the case that this is the simplest method to achieve your use case. If that’s true then by all means go for it. However, I suspect things can be simplified through the use of ZCAP-LD. ZCAP-LD is built on the concept of an object capabilities model which is well suited for authorization systems (but not for making claims about a subject). I’ll be writing up a follow-up blog on how the same use case described above could be designed with ZCAP-LD and show the differences between the two. Be on the lookout for that to come next!


Common Delegation Patterns in the Verifiable Credential Ecosystem

Here's 3 ways you can utilize VCs and DIDs to enable delegation and attenuated delegation for more complex scenarios.
Common Delegation Patterns in the Verifiable Credential Ecosystem

It’s commonly understood that verifiable credentials are a useful data model for expressing provenance and authority of data. In other words, they’re great for solving the “who says what” problem in a digital ecosystem. However, did you know that there are three ways in which you can utilize VCs and DIDs to enable delegation and attenuated delegation for more complex scenarios? In this blog post, I’ll cover the three patterns you can use with examples to help you figure out some of the more advanced capabilities of the VC data model. See below for more details!

Terminology

Since it’s quite common to see these terms within the IAM space, I figured it would be useful to first cover what each term means in simple term.

Delegation: This is the ability for someone to share their abilities with another user within the system. There’s multiple ways this can be done all with different forms of tradeoffs.

A user who shares their password with another user is performing delegation, but not in a way that allows an authorization endpoint (often times called a “verifier”) to be able to uniquely differentiate between the two users. This inability to differentiate at the authorization endpoint often leads to concerns around the confused deputy problem.

This when the system has been designed to allow the user to share their abilities with another user. The most common way this shows up today is with google document links. When a user “allows anyone with this link to view/comment/edit” they’re granting abilites to other users in a delegated way. The difference being the system can identify these unique users, which is commonly seen via the further delegation based upon an email address or organization.

Attenuated delegation: This is when the user opts to share only a portion of the abilities that they have. So looking at our options from delegation, option 1 would not allow this because the user is sharing all the same abilities that they have when logging into an account. So if an admin shares their password, the person they share the password with has admin abilities as well. One of the more common examples I can think of when attenuated delegation that is in use today is when a valet key is given to a valet. This key gives the valet access to drive the vehicle, but doesn’t allow them to open a glovebox or trunk for example. In this case, the driver of the vehichle is able to share only a portion of their abilities (driving) without sharing all of their abilities (opening a glovebox or trunk). In almost every system I’ve seen, this has to be intentionally designed into the system.

Delegation without attenuation by using DID Documents

As an example, let’s say we have the two following DID Documents:

Alice’s DID Document:

{ "@context": [ "https://www.w3.org/ns/did/v1", "https://w3id.org/security/suites/ed25519-2020/v1" ], "id": "did:example:alice", "verificationMethod": [ { "id": "did:example:alice#aliceKey1", "type": "Ed25519VerificationKey2020", "controller": "did:example:alice", "publicKeyMultibase": "zH3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV" }, { "id": "did:example:bob#bobKey1", "type": "Ed25519VerificationKey2020", "controller": "did:example:bob", "publicKeyMultibase": "z9hFgmPVfmBZwRvFEyniQDBkz9LmV7gDEqytWyGZLmDXE" } ], "assertionMethod": [ "did:example:alice#aliceKey1", "did:example:bob#bobKey1" ] }

Bob’s DID Document:

{ "@context": [ "https://www.w3.org/ns/did/v1", "https://w3id.org/security/suites/ed25519-2020/v1" ], "id": "did:example:bob", "verificationMethod": [ { "id": "did:example:bob#bobKey1", "type": "Ed25519VerificationKey2020", "controller": "did:example:bob", "publicKeyMultibase": "z9hFgmPVfmBZwRvFEyniQDBkz9LmV7gDEqytWyGZLmDXE" } ], "assertionMethod": ["did:example:bob#bobKey1"] }

In this example we can see that Alice has the capability to delegate to Bob the ability for him to assert on her behalf. In the case of the following VC, bob could then create a valid verifiable presentation based on the following verifiable credential that was delegated to him:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", "https://www.w3.org/2018/credentials/examples/v1" ], "id": "http://example.gov/credentials/3732", "type": ["VerifiableCredential", "UniversityDegreeCredential"], "issuer": { "id": "did:example:issuer" }, "issuanceDate": "2020-03-10T04:24:12.164Z", "credentialSubject": { "id": "did:example:alice", "degree": { "type": "BachelorDegree", "name": "Bachelor of Science and Arts" } }, "proof": { "type": "JsonWebSignature2020", "created": "2020-03-21T17:51:48Z", "verificationMethod": "did:example:issuer#credentialIssuanceKey", "proofPurpose": "assertionMethod", "jws": "eyJiNjQiOmZhbHNlLCJjcml0IjpbImI2NCJdLCJhbGciOiJFZERTQSJ9..OPxskX37SK0FhmYygDk-S4csY_gNhCUgSOAaXFXDTZx86CmI5nU9xkqtLWg-f4cqkigKDdMVdtIqWAvaYx2JBA" } }

Where the verifiablePresentation looks like so:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", "https://example.com/credentials/latest" ], "type": ["VerifiablePresentation"], "verifiableCredential": [ { "@context": [ "https://www.w3.org/2018/credentials/v1", "https://www.w3.org/2018/credentials/examples/v1" ], "id": "http://example.gov/credentials/3732", "type": ["VerifiableCredential", "UniversityDegreeCredential"], "issuer": { "id": "did:example:issuer" }, "issuanceDate": "2020-03-10T04:24:12.164Z", "credentialSubject": { "id": "did:example:alice", "degree": { "type": "BachelorDegree", "name": "Bachelor of Science and Arts" } }, "proof": { "type": "JsonWebSignature2020", "created": "2020-03-21T17:51:48Z", "verificationMethod": "did:example:issuer#credentialIssuanceKey", "proofPurpose": "assertionMethod", "jws": "eyJiNjQiOmZhbHNlLCJjcml0IjpbImI2NCJdLCJhbGciOiJFZERTQSJ9..OPxskX37SK0FhmYygDk-S4csY_gNhCUgSOAaXFXDTZx86CmI5nU9xkqtLWg-f4cqkigKDdMVdtIqWAvaYx2JBA" } } ], "id": "ebc6f1c2", "holder": "did:example:alice", "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "verificationMethod": "did:example:bob#bobKey1", "proofPurpose": "authentication", "challenge": "123", "proofValue": "z5LgJQhEvrLoNqXSbBzFR6mqmBnUefxX6dBjn2A4FYmmtB3EcWC41RmvHARgHwZyuMkR9xMbMCY7Ch4iRr9R8o1JffWY63FRfX3em8f3avb1CU6FaxiMjZdNegc" } }

This effectively is a method to allow Alice to add bob’s public key to her did document and giving him a copy of the verifiable credential to delegate him to act on her behalf. However, because this pattern relies on Alice granting Bob full authority to perform any action on her behalf, it lacks attenuation, but does have delegation.

Delegation by VCs

Additionally, there’s a well documented case to be able to establish delegation using just VCs without the ability to attenuate the data. See Section Appendix C.5 for more details on this pattern.

Attenuated Delegation by VCs

Further, by combining the pattern described in Section Appendix C.5 and combining it with the usage of a selective disclosure scheme like BBS signatures, we can enable an attenuated delegation pattern.

This is done via modifying the type of proof used to sign the credential and then utilizing the selective disclosure to limit the data. Using the following original verifiable credential:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", "https://w3id.org/citizenship/v1", "https://w3id.org/security/bbs/v1" ], "id": "https://issuer.oidp.uscis.gov/credentials/83627465", "type": ["VerifiableCredential", "PermanentResidentCard"], "issuer": "did:example:issuer", "identifier": "83627465", "name": "Permanent Resident Card", "description": "Government of Example Permanent Resident Card.", "issuanceDate": "2019-12-03T12:19:52Z", "expirationDate": "2029-12-03T12:19:52Z", "credentialSubject": { "id": "did:example:b34ca6cd37bbf23", "type": ["PermanentResident", "Person"], "givenName": "Alice", "familyName": "SMITH", "gender": "Female", "image": "data:image/png;base64,iVBORw0KGgokJggg==", "residentSince": "2015-01-01", "lprCategory": "C09", "lprNumber": "999-999-999", "commuterClassification": "C1", "birthCountry": "Bahamas", "birthDate": "1958-07-17" }, "proof": { "type": "BbsBlsSignature2020", "created": "2020-10-16T23:59:31Z", "proofPurpose": "assertionMethod", "proofValue": "kAkloZSlK79ARnlx54tPqmQyy6G7/36xU/LZgrdVmCqqI9M0muKLxkaHNsgVDBBvYp85VT3uouLFSXPMr7Stjgq62+OCunba7bNdGfhM/FUsx9zpfRtw7jeE182CN1cZakOoSVsQz61c16zQikXM3w==", "verificationMethod": "did:example:issuer#test" } }

Then the following derived proof could be used to attenuate the data while also delegating the credential capabilities to a new holder:

{ "@context": [ "https://www.w3.org/2018/credentials/v1", "https://www.w3.org/2018/credentials/examples/v1" ], "id": "did:example:76e12ec21ebhyu1f712ebc6f1z2", "type": ["VerifiablePresentation"], "verifiableCredential": [ { "@context": [ "https://www.w3.org/2018/credentials/v1", "https://w3id.org/citizenship/v1", "https://w3id.org/security/bbs/v1" ], "id": "https://issuer.oidp.uscis.gov/credentials/83627465", "type": ["PermanentResidentCard", "VerifiableCredential"], "description": "Government of Example Permanent Resident Card.", "identifier": "83627465", "name": "Permanent Resident Card", "credentialSubject": { "id": "did:example:alice", "type": ["Person", "PermanentResident"], "familyName": "SMITH", "gender": "Female", "givenName": "Alice" }, "expirationDate": "2029-12-03T12:19:52Z", "issuanceDate": "2019-12-03T12:19:52Z", "issuer": "did:example:issuer", "proof": { "type": "BbsBlsSignatureProof2020", "nonce": "wrmPiSRm+iBqnGBXz+/37LLYRZWirGgIORKHIkrgWVnHtb4fDe/4ZPZaZ+/RwGVJYYY=", "proofValue": "ABkB/wbvt6213E9eJ+aRGbdG1IIQtx+IdAXALLNg2a5ENSGOIBxRGSoArKXwD/diieDWG6+0q8CWh7CViUqOOdEhYp/DonzmjoWbWECalE6x/qtyBeE7W9TJTXyK/yW6JKSKPz2ht4J0XLV84DZrxMF4HMrY7rFHvdE4xV7ULeC9vNmAmwYAqJfNwY94FG2erg2K2cg0AAAAdLfutjMuBO0JnrlRW6O6TheATv0xZZHP9kf1AYqPaxsYg0bq2XYzkp+tzMBq1rH3tgAAAAIDTzuPazvFHijdzuAgYg+Sg0ziF+Gw5Bz8r2cuvuSg1yKWqW1dM5GhGn6SZUpczTXuZuKGlo4cZrwbIg9wf4lBs3kQwWULRtQUXki9izmznt4Go98X/ElOguLLum4S78Gehe1ql6CXD1zS5PiDXjDzAAAACWz/sbigWpPmUqNA8YUczOuzBUvzmkpjVyL9aqf1e7rSZmN8CNa6dTGOzgKYgDGoIbSQR8EN8Ld7kpTIAdi4YvNZwEYlda/BR6oSrFCquafz7s/jeXyOYMsiVC53Zls9KEg64tG7n90XuZOyMk9RAdcxYRGligbFuG2Ap+rQ+rrELJaW7DWwFEI6cRnitZo6aS0hHmiOKKtJyA7KFbx27nBGd2y3JCvgYO6VUROQ//t3F4aRVI1U53e5N3MU+lt9GmFeL+Kv+2zV1WssScO0ZImDGDOvjDs1shnNSjIJ0RBNAo2YzhFKh3ExWd9WbiZ2/USSyomaSK4EzdTDqi2JCGdqS7IpooKSX/1Dp4K+d8HhPLGNLX4yfMoG9SnRfRQZZQ==", "verificationMethod": "did:example:issuer#test", "proofPurpose": "assertionMethod", "created": "2020-10-16T23:59:31Z" } }, { "@context": [ "https://www.w3.org/2018/credentials/v1", "https://w3id.org/citizenship/v1", "https://w3id.org/security/v3-unstable" ], "id": "https://issuer.oidp.uscis.gov/credentials/83627465", "type": ["PermanentResidentCard", "VerifiableCredential"], "description": "Government of Example Permanent Resident Card.", "identifier": "83627465", "name": "Permanent Resident Card", "credentialSubject": { "id": "did:example:bob", "type": ["Person", "PermanentResident"], "familyName": "SMITH", "gender": "Female", "givenName": "Alice" }, "expirationDate": "2029-12-03T12:19:52Z", "issuanceDate": "2019-12-03T12:19:52Z", "issuer": "did:example:alice", "proof": { "type": "Ed25519Signature2020", "created": "2019-12-11T03:50:55Z", "proofValue": "z5LgmVhjjPTEzGL31k2eEde8bdr4MAzxQv87AmdHt5Usd1uGK1Ae88NoZ5jgTLKS6sJCZnQNthR3qAbyRMxvkqSkss2WtyKLa9rqhJmR6YEBkiuUtxawhrscWXm", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:alice#aliceKey1" } } ], "proof": [ { "type": "Ed25519Signature2020", "created": "2018-06-18T21:19:10Z", "proofPurpose": "assertionMethod", "verificationMethod": "did:example:bob#bobKey1", "challenge": "c0ae1c8e-c7e7-469f-b252-86e6a0e7387e", "jws": "BavEll0/I1..W3JT24=" } ] } Conclusion

So there’s a few different take aways from this that should be highlighted. First and foremost VCs and DIDs have some really interesting capabilities that enable delegation and attenuated delegation. However, it’s not always advantageous and requires thorough understanding of the use case that’s trying to be built with DIDs and VCs and whether or not delegation needs to be enabled or disabled. In some cases, it’s being left in when there’s an expectation that a credential is only able to be used by a single holder. Finally, it should be noted that while VCs are capable of being used for authorization systems, it’s generally not a good idea to do so. This is because the complexity (based on the number of checks) of the authorization system is quite high and the semantics have not been designed in a way that align well with most authorization systems meaning a vulnerability is more likely to be introduced. If you’re looking for these capabilities, I would recommend you look to the ZCAP-LD data model which is designed especially for these concepts. And if you’re still confused and would like some help please reach out and I can see how I can help.

Monday, 21. June 2021

Phil Windley's Technometria

Reciprocal Negotiated Accountability

Summary: The self-sovereign internet underscores a tension between those who desire perfect oversight of society and those who desire perfect privacy. In this post, I explore a method for conditional legibility of private communications. In Self-Sovereign Communication, Oskar Van Deventer, discusses the communications layer enabled by DIDs. This is the same layer that I've labeled t

Summary: The self-sovereign internet underscores a tension between those who desire perfect oversight of society and those who desire perfect privacy. In this post, I explore a method for conditional legibility of private communications.

In Self-Sovereign Communication, Oskar Van Deventer, discusses the communications layer enabled by DIDs. This is the same layer that I've labeled the self-sovereign internet.

Oskar lays out nine requirements for self-sovereign communications (emphasis added):

The communication channel shall be usable for machine-readable issuer-holder-verifier interactions . The communication channel shall be protected against eavesdropping, impersonation, message modification and repudiation. Parties shall be able to digitally find each other and to establish a communication channel. The communication channel between counterparties shall be persistent. The communication channel shall be intrinsically symmetrical. The communication channel shall not unnecessarily disclose information between counterparties or to third parties. The communication channel shall be unilaterally closable. The communication channel shall not depend on third parties more than needed. The communication channel shall enable compliance with legal requirements, like legal intercept.

I was pleased to see these principles laid out clearly because many of them are often discussed (including by me) as properties of DIDComm, without the precision Oskar imposes.

The last, as Oskar concedes, is likely to be the most controversial. Indeed, when I read it my first reaction was to start arguing. If complying with legal requirements means creating backdoors to DIDComm, I'd oppose it.

The problem with backdoors for complying with legal requirements is that now developers and cloud operators are left with the task of determining who the good guys are. The whole point of decentralized communication systems is to avoid the kind of centralized, single-point-of-failure that backdoors imply.

Reciprocal Negotiated Accountability

In Reciprocal Negotiated Accountability, Daniel Hardman proposes an alternative to backdoors.

Daniel's idea is to combine two capabilities to create a decentralized system for enabling accountability.

The first is digital watermarks and data terms of service. The watermark is a cryptographically signed addition to the original document that states the terms behind the sharing. For example, a sales agreement could include data sharing terms that state the recipient may not disclose named aspects of the document except under legal subpoena.

The second is provisional anonymity where identifying information is encrypted and the encrypted packaged is shared with the recipient. The keys to decrypt the identifying information are shared with a third party under escrow with legal requirements that the keys only be reveled to the recipient under specific conditions.

Daniel combines these into a decentralized system of opt-in agreements between parties that are tailored to the context and circumstances of the specific communications channel and data sharing. The legal agreement defines the requirements that must be met for access.

Daniel calls this "reciprocal negotiated accountability" because both parties negotiate an agreement about how shared data will be treated.

Daniel's solution won't make those who wish for unfettered access to communications channels happy. But it represents an alternative to backdoors that solves many of the problems backdoors present while protecting privacy for legitimate uses–as negotiated by the parties sharing data.

Photo Credit: 3D Tin Can Phones from Chris Potter (CC BY 2.0)

Tags: identity ssi didcomm privacy cryptography


The Self-Sovereign Internet

Summary: I'm attended the Hyperledger Global Forum this last week–all virtual. I spoke on DIDComm and the Self-Sovereign Internet. If you're not up to speed with DIDComm and how it creates a secure overlay network on TCP/IP, you should be. I consider it one of the most important things to come out of the SSI movement. DIDs and DID-based Communication or DIDComm form the second layer of the SSI

Summary: I'm attended the Hyperledger Global Forum this last week–all virtual. I spoke on DIDComm and the Self-Sovereign Internet. If you're not up to speed with DIDComm and how it creates a secure overlay network on TCP/IP, you should be. I consider it one of the most important things to come out of the SSI movement.

DIDs and DID-based Communication or DIDComm form the second layer of the SSI stack, providing a secure communications layer for the exchange of identity information via verifiable credentials. But, because of it's flexibility and the ability to define protocols on top of DIDComm messaging, it promises to be as important as the identity layer it enables. Autonomic Identifiers

The foundation of the self-sovereign internet is built on autonomic identifiers. I'm going to speak about Peer DIDs here, but KERI and other systems provide autonomic identifiers that serve just as well.

Identity systems provide the means to remember, recognize, and rely on other parties in a relationship. To do so, they use identifiers, convenient handles that name the thing being remembered.

Identifiers are issued to or created by a controller (e.g. Alice) who, by virtue of knowing the authentication factors (e.g. password, key fob, cryptographic key), can make authoritative statements about the identifier (e.g. claim it by logging in).

Bindings between Alice, a Peer DID, and the public key it points to. (click to enlarge)

In an autonomic identity architecture, the controller, Alice, generates a public-private key pair, derives a globally unique identifier, and shares the identifier and the currently associated public key with others to create relationships. Alice uses her private key to sign statements that authenticate herself and authorize use of the identifier. A digital signature also provides the means for Alice to cryptographically respond to challenges so she can prove she controls the identifier. These self-authentication and self-authorization capabilities make the identifier self-certifying and self-managing, meaning that there is no external third party, not even a ledger, needed for Alice to manage and use the identifier and prove to others the integrity of the bindings between herself and the identifier.

Any entity can create and establish control over an identifier in a manner that is independent, interoperable, and portable without recourse to any central authority. Autonomic identity systems rely solely on self-sovereign authority.

Alice can create as many Peer DIDs as she needs, each pointing to a public key that Alice controls. Alice can rotate the keys underneath the Peer DID anytime without impacting the identifier or her ability to prove she controls it. She keeps track of these key events in a key event log. The key event log is a chain of signed change records that can be cryptographically verified. Alice can use it to prove the provenance of her control of the identifier from its inception to the present.

Peer DID Exchange

Alice can exchange Peer DIDs with Bob to create a relationship. Because DIDs are associated with public keys, the exchange ensures that Alice and Bob have each other's public keys. They share key event logs (using a CRDT) for each identifier. If either updates the keys associated with the DID, the other is informed of change.

Alice and Bob exchange Peer DIDs to create a relationship (click to enlarge)

Having exchanged DIDs, Alice and Bob can now exchange signed and encrypted messages with each other using DIDComm. DIDComm is a messaging protocol that rides on top of this Peer DID relationship.

Alice and Bob use DIDComm to exchange messages (click to enlarge)

Alice and Bob are using digital wallets to store the keys (and manage their key event logs). The DIDComm messages are being exchanged using software agents that understand the DIDComm messaging protocol and use the keys in the wallet.

Alice can have DID-based relationships with multiple people, organizations, and even things. Each relationship includes a secure DIDComm-based messaging capability.

Some of Alice's Relationships (click to enlarge)

This network of DID-based relationships forms an overlay network. An overlay network comprises virtual links that correspond to a path in the underlying network. Secure overlay networks rely on an identity layer based on asymmetric key cryptography to ensure message integrity, non-repudiation, and confidentiality.

DIDComm messaging has several important properties that, taken together, provide a generative, secure network overlay for the Internet.

Secure - DID-based relationships are mutually authenticating. Private - messages can be encrypted. Interoperable - messages can be exchanged between any agents that support the DIDComm protocol. Transport-agnostic - DIDComm does not rely on any specific network technology–it is as happy on Bluetooth as on TCP/IP or anything else. Extensible - DIDComm is designed to support other protocols riding on top of its general secure messaging infrastructure. Protocological Power

The extensibility of DIDComm is one of its most powerful features because it makes DIDComm generative–just like the Internet itself.

Protocols describe the rules for a set of interactions, specifying the kinds of interactions that can happen without being overly prescriptive about their nature or content. Protocols formalize workflows for specific interactions like ordering food at a restaurant, playing a game, or applying for college.

The Hyperledger Aries project has a collection of RFCs that describe protocols for DIDComm messaging. While we have come to think of SSI agents being strictly about exchanging peer DIDs to create a connection, request and issue a credential, or prove things using credentials, these are merely specific protocols defined to run over the DIDComm messaging protocol. The follow specifications describe the protocols for these three core applications of DIDComm:

Connecting with others Requesting and issuing credentials Proving things using credentials

Dozens, even hundreds, of other protocols are possible.

Daniel Hardman has provided a comprehensive tutorial on defining protocols on DIDComm. One of the Aries RFCs is sample protocol definition of a protocol for playing TicTacToe over DIDComm messaging.

Alice and Bob play Tic Tac Toe (click to enlarge)

The TicTacToe protocol defines types of messages that are allowed, the game state, and what messages are allowed in each game state. I recommend it as a way to understand DIDComm protocols since it's familiar and easy to understand. Bruce Conrad who works on picos with me implemented the TicTacToe protocol for picos, which act as DIDComm agents.

Generativity

In 2005, Jonathan Zittrain wrote a compelling and prescient examination of the generative capacity of the Internet and its tens of millions of attached PCs. Zittrain defined generativity thus:

Generativity denotes a technology's overall capacity to produce unprompted change driven by large, varied, and uncoordinated audiences. From The Generative Internet
Referenced 2021-06-14T13:41:18-0600

Generative systems use a few basic rules, structures, or features to yield behaviors that can be extremely varied and unpredictable. Zittrain goes on to lay out the criteria for evaluating the generativity of a technology:

Generativity is a function of a technology's capacity for leverage across a range of tasks, adaptability to a range of different tasks, ease of mastery, and accessibility.

I have made the case elsewhere that the self-sovereign internet meets Zittrain's criteria for generativity.

Generativity provides decentralized actors with the ability to create cooperating, complex structures and behavior. No one person or group can or will think of all the possible uses, but each is free to adapt the system to their own use. The architecture of the self-sovereign internet enables adaptation of DIDComm messaging to any circumstance.

I am bullish on the possibilities for verifiable credentials to allow people to live digital lives with dignity and effectiveness, addresses the problems of social inclusion, and support economic equality to everyone around the globe. With all that, I believe the possibilities for the self-sovereign internet are even larger, promising a more secure and private, albeit no less useful, internet for tomorrow. DIDComm may turn out to be the most important part of self-sovereign identity.

Tags: ssi identity didcomm ssiot

Sunday, 20. June 2021

reb00ted

Everybody has our data ... except us

Google has all my e-mail. (And I don’t. They merely let me access it with a browser.) Facebook has the list of all of my friends and what I said to them. (And I don’t.) LinkedIn has all of my business contacts. (Repeat after me: and I don’t.) Instagram has all my photos. Well, the Instagram department of Facebook does. (Chorus now: and I don’t.) Amazon has the list of all my purchases, an

Google has all my e-mail. (And I don’t. They merely let me access it with a browser.)

Facebook has the list of all of my friends and what I said to them. (And I don’t.)

LinkedIn has all of my business contacts. (Repeat after me: and I don’t.)

Instagram has all my photos. Well, the Instagram department of Facebook does. (Chorus now: and I don’t.)

Amazon has the list of all my purchases, and knows what products I was interested in but didn’t buy after all. (AND I DON’T.)

The list goes on.

Does this sound right to you? It sounds quite wrong to me.

But maybe it doesn’t matter. Things go well in this arrangement, don’t they?

Not so fast. Let’s start with losing access to my accounts. Maybe I forgot my password and failed to reset it. Or maybe I managed to get a password reset e-mail but my primary e-mail account was hacked, and now the attacker controls my Facebook account and I don’t. Maybe Google decided to disable my account, and given that there is no appeals process, that’s the end of me and my e-mail. The end of a 20-year-long record of all important things I wrote to others and they wrote to me. In the blink of an eye, because they have my data and I don’t.

But even if everything is fine, and I don’t forget my passwords and don’t get hacked and won’t get locked out: it’s a bit like being a really unimportant guest living in a fancy hotel on a short-term lease. They can kick me out any time, and keep all my furniture and other possessions, no questions asked. Thank you, I prefer my very own home, where nobody can kick me out, or at least renter protection laws, which don’t exist online.

We got to get our data back.


SSI Ambassador

Digital Identity Wallet: A place for your self-sovereign identity

Illustration: Digital Identity Wallet This article explains what a wallet is, how it works and how you can use it for managing your digital identity. While there are also wallets for institutions, this article only focuses on wallets for end users. The article also takes into consideration the revised eIDAS regulation including the ‘European Digital Identity Wallet’. An Introduction A d
Illustration: Digital Identity Wallet

This article explains what a wallet is, how it works and how you can use it for managing your digital identity. While there are also wallets for institutions, this article only focuses on wallets for end users. The article also takes into consideration the revised eIDAS regulation including the ‘European Digital Identity Wallet’.

An Introduction

A digital wallet is a key management application, which provides a user with a graphical interface to store, manage and secure digital keys. These keys can be used to sign transactions, statements, credentials, documents or claims.

A digital identity wallet enables a user to establish relationships and interact with third parties in a trusted manner. While the wallet aspect is mainly dealing with key management, storage aspects and the graphical interface (UI = User Interface), the third party interactions are rather organised by your agent, which is a part of your wallet. Your agent handles third-party interactions in your name and acts in your interest. It is a piece of software, which helps you to stay in control of your messaging, security, health records, privacy, purchases etc.

A digital identity wallet based on self-sovereign identity (SSI) principles.

Not all wallets are the same! A wallet is a piece of technology, which can be implemented in different ways. While it can leverage decentralised identifiers (DIDs) and verified credentials, it doesn’t necessarily need to follow the principles of SSI. There are cloud wallets, which are hosted and provided by a third party. A good comparison is a wallet for cryptocurrencies — think of an exchange like Coinbase or Binance. These companies offer you a hosted wallet, which you don’t control. You might have the authentication means (password + second factor) to access your wallet, but it’s not yours. In this case, you are the owner, but not the possessor and you don’t have data sovereignty. This is in stark contrast to so-called self-custody wallets, which you install on your device. These wallets randomly create a private key, which only you know. The key point is that your wallet creates keys for you, which only you know and not a third party.

A digital identity wallet, which follows the principles of SSI enables the user to have data sovereignty and complete control as well as data portability. It provides you with the necessary autonomy to be independent of a third party. It not only enables ownership but also possession. It enables transparency and explicit consent when sharing information. It’s also vendor-independent, meaning you can export your data and import it into another wallet of your choice since it is built on open standards, which are widely used. It also creates peer identifiers for every new contact or rather an interaction, instead of getting an identifier assigned to it.

The core functions of an SSI wallet

Please note that the described characteristics below are not universally valid for all (SSI) wallets. It’s rather a desired status.

Interactions in an SSI ecosystem, picture by Adrian Doerk from Lissi.

Establish trusted relationships with third parties:
The wallet can create an encrypted communication channel to exchange information between you and a (trusted) third party. This communication channel is based on a unique identifier, which you control. Hence it is not hosted by a third party. This in turn enables you the portability of communication channels. Meaning if you switch from one wallet to another you can use the communication channels, which you already created without the reliance on any third-party platform.

Store, organise and present verified data:
The user can store and manage (verified) credentials among other information within the wallet. Once in the wallet, credentials can be used to answer a proof request from every connection. The wallet creates a verifiable presentation, which the user can choose to send or instead decline the proof request. Users are also able to verify the identity of the other party, effectively establishing a trusted relationship, which can be leveraged to share and receive information within a defined trust framework such as the eIDAS regulation. This trust might be based on an electronic seal or similar trust mechanisms. However, this might not be the case for all contacts.

The information exchanged can be verified, but can also be self-attested or just proof a certain threshold without revealing the exact information like your age when proofing you are 18 or older.

A transparent history of shared data:
Since the wallet usually keeps a history of interactions, the user can track who shared what data and when. This leads to greater transparency for the user and helps to better exercise data protection rights. A framework for the integration of detailed consent forms is currently under development.

Self-custody: With great control comes great responsibility

These digital wallets run locally as an application on the device of the user. From a technical perspective, these wallets are similar to self-custody wallets for cryptocurrencies. Similar to these wallets the user has the responsibility to do a backup.

Identification, authentication and authorization:
Before diving deeper into the core functions of a digital wallet we need to understand the differences between the three words above. The questions are from the perspective of the verifier or issuer.

Identification answers the question: “Who is I’m talking to?”
Authentication answers the question: “Is it you (whom I’ve identified already) again?”
Authorization answers the question: “What rights do I want to grant you?”

The wallet can enable a variety of additional functions and thus serves as a central point for the user to manage and access services. For example, the wallet can be used to replace traditional authentication methods such as passwords with a single sign-on (SSO) functionality. Furthermore, existing standards such as the OpenID Connect protocol can also be connected to enable communication with existing infrastructure. Hence, once widespread adopted the wallet will completely replace passwords for you and enable you to identify and authenticate yourself, identity third-parties as well as authorise third-parties to use your data according to your permissions.

The European Digital Identity Wallet Illustration: European Digital Identity Wallet

The main regulation within the European Union, which addresses the topic of identification (among other topics) is the regulation on electronic identification and trust services for electronic transactions in the internal market better known as eIDAS regulation. The regulation just went through a major revision and now includes several aspects regarding self-sovereign identities in its draft version. It also includes the aspect of a European Digital Identity Wallet, which “is a product and service that allows the user to store identity data, credentials and attributes linked to her/his identity, to provide them to relying parties on request and to use them for authentication, online and offline, for a service in accordance with Article 6a; and to create qualified electronic signatures and seals”.

In the following, the European Digital Identity Wallet is referred to as “EU Wallet” for simplicity.

Functions of the EU Wallet:

It should technically enable the selective disclosure of attributes to relying parties. Member States should also reach an agreement on common elements of a business model and fee structure of the EU Wallets. The EU Wallet will furthermore enable citizens to create qualified electronic signatures that can facilitate political participation among other functions.

EU Wallets shall enable the user to: securely request and obtain, store, select, combine and share, in a manner that is transparent to and traceable by the user, the necessary legal personal identification data and electronic attestation of attributes to authenticate online and offline to use online public and private services; sign statements or contracts utilising qualified electronic signatures (QES) have full control of the EU Wallet. The issuer of the EU Wallet shall not collect information about the use of the EU Wallet which are not necessary for the provision of the EU Wallet service. use it free of charge (natural persons). access the wallet for persons with disabilities. The EU Wallets shall provide a common interface: to qualified and non-qualified trust service providers issuing qualified and non-qualified electronic attestations of attributes. for relying parties to request and validate personal identification data and electronic attestations of attributes. for the presentation to relying parties of personal identification data, electronic attestation of attributes or other data such as credentials, in local mode not requiring internet access for the wallet. Furthermore, the EU Wallets need to: ensure that trust service providers of qualified attestations of attributes cannot receive any information about the use of these attributes. meet the requirements in regards to assurance level “high”, in particular as applied to the requirements for identity proofing and verification, and electronic identification means management and authentication. provide a mechanism to ensure that the relying party is able to authenticate the user and to receive electronic attestations of attributes; ensure that the personal identification data uniquely and persistently represent the natural or legal person is associated with it. An EU Wallet shall be issued:

(a) by a Member State;
(b) under a mandate from a Member State;
(c) independently but recognised by a Member State.

The draft regulation requires Member States to issue a EU Wallet under a notified eID scheme to common technical standards following a compulsory compliance assessment and voluntary certification within the European cybersecurity certification framework, as established by the Cybersecurity Act.

Service providers should communicate their intent to rely on the EU Wallets to the Member States. Relying on the level of assurance “high”, the EU Wallets should benefit from the potential offered by tamper-proof solutions such as secure elements.

The conformity of EU Wallets should be certified by accredited public or private sector bodies designated by Member States. Certification should in particular rely on the relevant European cybersecurity certifications schemes. The EU Wallet Trust Mark’ means an indication in a simple, recognisable and clear manner that a EU Wallet has been issued in accordance with this Regulation.

The Commission shall establish, publish and maintain a list of certified European Digital Identity Wallets.

Where are heading? A personal opinion: A wallet acts as a central point to manage interactions in a digital world. It will be the standard for identity management and be ubiquitous in everyday life. Technology is moving fast. While standardisation is taking quite a while, institutions and governments have recognised the need for innovation in this area to secure data sovereignty and not be dependent on a foreign oligopoly of companies with irresponsible business practices. Dozens of use-cases from different industries are explored in parallel and will be available in the upcoming months. Over time digital wallets for pure identity management and financial applications (e.g. cryptocurrencies or central bank digital currencies (CBDC)) are likely to merge. The draft of the revised eIDAS regulation of the European Union is the most comprehensive and ambitious step towards SSI internationally. While the exact implementation and technical specifications of the EU Wallets are still to be determined, their compulsory issuance for the EU Member States as well as the compulsory acceptance for big platform providers will have a tremendous international impact. Key management is an issue, which still needs to be solved. This means, the user also has the responsibility of having a secure backup solution to restore the wallet in case of lost access. Sources used: Amending Regulation (EU) No 910/2014 as regards establishing a framework for a European digital Identity (eIDAS Draft). The current and future state of digital wallets, Darrell O’ Donnell, Making sense of digital Wallets, Digital Identification and Authentication Council of Canada (DIACC) Digital wallet and digital agents, Self-sovereign identity, manning publications What is a wallet, Kaliya Young What goes in a wallet, Daniel Hardmann, W3C CCG Security, Siloes and sovereignty, Daniel Hardman About the author:

Adrian Doerk is a trained IT-System Electrician and has a degree in international business. He focuses on data sovereignty for individuals, institutions and governments. Currently, he works at main incubator where he leads the business development for Lissi and is also responsible for the communication at IDunion. He’s also active in foundations such as DIF and ToIP. SSI Ambassador is a private educational endeavour.

Disclaimer:

This article does not represent the official view of any entity, which is mentioned in this article or which is affiliated with the author. It solely represents the opinion of the author.

Saturday, 19. June 2021

Doc Searls Weblog

Redux 001: BuzzPhrasing

Since I’m done with fighting in the red ocean of the surveillance-dominated Web, I’ve decided, while busy working in the blue ocean (on what for now we’re calling i-commerce), to bring back, in this blog, some of the hundreds of things I’ve written over the last 30+ years. I’m calling it the Redux series. To qualify, […]

Since I’m done with fighting in the red ocean of the surveillance-dominated Web, I’ve decided, while busy working in the blue ocean (on what for now we’re calling i-commerce), to bring back, in this blog, some of the hundreds of things I’ve written over the last 30+ years. I’m calling it the Redux series. To qualify, these should still ring true today, or at least provide some history. This early one is still on the Web, here at BuzzPhraser.com. I’ve made only two small edits, regarding dates. (And thanks to Denise Caruso for reminding me that this thing started out on paper, very long ago.)

The original BuzzPhraser was created in 1990, or perhaps earlier, as a spreadsheet, then a HyperCard stack; and it quickly became one of the most-downloaded files on AOL and Compuserve. For years after that it languished, mostly because I didn’t want to re-write the software. But when the Web came along, I knew I had to find a way to re-create it. The means didn’t find that end, however, until Charles Roth grabbed the buzzwords by their serifs and made it happen, using a bit of clever Javascript. Once you start having fun with the new BuzzPhraser, I’m sure you’ll thank him as much as I do.

The story that follows was written for the original BuzzPhraser. I thought it would be fun to publish it unchanged.

—Doc, sometime in the late ’90s

BuzzPhrases are built with TechnoLatin, a non-language that replaces plain English nouns with vague but precise-sounding substitutes.  In TechnoLatin, a disk drive is a “data management solution.”  A network is a “workgroup productivity platform.”  A phone is a “telecommunications device”.

The virtue of TechnoLatin is that it describes just about anything technical.  The vice of TechnoLatin is that it really doesn’t mean anything.  This is because TechnoLatin is comprised of words that are either meaningless or have been reduced to that state by frequent use.  Like the blank tiles in Scrabble, you can put them anywhere, but they have no value.  The real value of TechnoLatin is that it sounds precise while what it says is vague as air.  And as easily inflated.

Thanks to TechnoLatin, today’s technology companies no longer make chips, boards, computers, monitors or printers.  They don’t even make products.  Today everybody makes “solutions” that are described as “interoperable,” “committed,” “architected,” “seamless” or whatever.  While these words sound specific, they describe almost nothing.  But where they fail as description they succeed as camouflage: they conceal meaning, vanish into surroundings and tend to go unnoticed.

Take the most over-used word in TechnoLatin today: solution.  What the hell does “solution” really mean?  Well, if you lift the camouflage, you see it usually means “product.”  Try this: every time you run across “solution” in a technology context, substitute “product.”  Note that the two are completely interchangeable.  The difference is, “product” actually means something, while “solution” does not.  In fact, the popularity of “solution” owes to its lack of specificity.  While it presumably suggests the relief of some “problem,” it really serves only to distance what it labels from the most frightening risk of specificity: the clarity of actual limits.

The fact is, most vendors of technology products don’t like to admit that their creations are limited in any way.  Surely, a new spreadsheet — the labor of many nerd/years — is something more than “just a spreadsheet.”  But what?  Lacking an available noun, it’s easy to build a suitable substitute with TechnoLatin.  Call it an “executive information matrix.”  Or a “productivity enhancement engine.”  In all seriousness, many companies spend months at this exercise.  Or even years.  It’s incredible.

There is also a narcotic appeal to buzzphrasing in TechnoLatin.  It makes the abuser feel as if he or she is really saying something, while in fact the practice only mystifies the listener or reader.  And since buzzphrasing is so popular, it gives the abuser a soothing sense of conformity, like teenagers get when they speak slang.  But, like slang, TechnoLatin feels better than it looks.  In truth, it looks suspicious.  And with good reason.  TechnoLatin often does not mean what it says, because the elaborate buzzphrases it builds are still only approximations.

But who cares? Buzzphrasing is epidemic.  You can’t get away from it.  Everybody does it.  There is one nice thing about Everybody, however: they’re a big market.

So, after studying this disease for many years, I decided, like any self-respecting doctor, to profit from the problem.  And, like any self-respecting Silicon Valley entrepreneur, I decided to do this with a new product for which there was absolutely no proven need, in complete faith that people would buy it.  Such is the nature of marketing in the technology business.

But, lacking the investment capital required to generate demand where none exists, I decided on a more generous approach: to give it away, in hope that even if I failed to halt the epidemic, at least I could get people to talk about it.

With this altruistic but slightly commercial goal in mind, I joined farces with Ray Miller of Turtlelips Services to create a product that would encourage and support the narcotic practice of buzzphrasing.  Being the brilliant programmer he is, Ray hacked it into a stack in less time than it took for me to write this prose.  And now here it is, free as flu, catching on all over the damn place.

What made BuzzPhraser possible as a product is that the practice of buzzphrasing actually has rules.  Like English, TechnoLatin is built around nouns.  It has adjectives to modify those nouns.  And adverbs to modify the adjectives.  It also has a class of nouns that modify other nouns — we call them “adnouns.”  And it has a nice assortment of hyphenated prefixes and suffixes (such as “multi-” and “-driven”) that we call “hyphixes.”

Since the TechnoLatin lexicon is filled with meaningless words in all those categories, the words that comprise TechnoLatin buzzphrases can be assembled in just about any number or order, held together as if by velcro.  These are the rules:

adverbs modify adjectives adjectives modify adnouns, nouns or each other adnouns modify nouns or other adnouns nouns are modified by adnouns or adjectives prefixes modify all adjectives suffixes qualify all adnouns

Here is a diagram that shows how the rules work:

As with English, there are many exceptions.  But, as with programming, we don’t make any.  So cope with it.

With one adverb, one adjective, two adnouns, a noun and a prefix, you get “backwardly architected hyper-intelligent analysis inference leader.”  With an adjective and two nouns, you get “interactive leverage module.”  Put together buzzphrases of almost any shape and length:

“Breakthrough-capable technology market” “Primarily distinguished optional contingency philosophy control power environment” “Executive inference server” “Evidently complete key business manipulation capacity method” “Incrementally intelligent workgroup process topology vendor”

The amazing thing is that all of these sound, as we say in TechnoLatin, “virtually credible.”  And one nice thing about the computer business is — thanks largely to the brain-softening results of prolonged TechnoLatin abuse — “virtually credible” is exactly what it means in plain English: close enough.

BuzzPhraser makes “close enough” easy to reach by substituting guesswork for thinking.  Just keep hitting the button until the right buzzphrase comes along.  Then use that buzzphrase in faith that at least it sounds like you know what you’re saying.  And hey, in this business, isn’t that virtually credible?

Acknowledgements

Thanks to:

Stewart Alsop II, who published “Random Strings of TechnoLatin” along with the original Generic Description Table in both the Preceedings and Proceedings of Agenda 90; and who would like an e-mail front end that automatically discards any message with too many TechnoLatin words and buzzphrases.

Spencer F. Katt of PC Week, who devoted parts of two consecutive rumor columns to the Table, and posted it on the magazine’s CompuServe bulletin board, from which so many people copied it that I thought there might be something going on here.

Guy Kawasaki, who told me “this needs to be a product.”

Bob LeVitus, who told me “you ought to get this hacked into a stack.”

And Ray Miller, who did it.  Beautifully.

Doc Searls
Palo Alto, California
March 7, 1991


Identity Woman

Fake Students

It became clear to me again today why we here in California need Verifiable Credentials. I teach in a CCC – a California Community College. This summer I have a class and right now is the “census deadline” to drop students who haven’t been attending class. Below is the note we were sent regarding fraudulent […] The post Fake Students appeared first on Identity Woman.

It became clear to me again today why we here in California need Verifiable Credentials. I teach in a CCC – a California Community College. This summer I have a class and right now is the “census deadline” to drop students who haven’t been attending class. Below is the note we were sent regarding fraudulent […]

The post Fake Students appeared first on Identity Woman.

Tuesday, 15. June 2021

Hyperonomy Digital Identity Lab

The Verifiable Economy: Fully Decentralized Object (FDO) Example: Bob’s UDID Document

Strongly-typed Code to Generate Bob’s UDID Document Bob’s UDID Document

Strongly-typed Code to Generate Bob’s UDID Document

Bob’s UDID Document

{ "CellId": 6601258412767401213, "CredentialCore": { "udid": "did:svrn:credential:FD54/8F2B/8D61/9C5B", "context": [ "https://www.sovrona.com/ns/svrn/v1" ], "claims": [ { "key": "authentication", "attribute": [ { "key": "publicKey", "value": "1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ" }, { "key": "id", "value": "#pubkey1" }, { "key": "type", "value": "AUTHN-KEY" } ] }, { "key": "service", "attribute": [ { "key": "serviceEndPoint", "value": "http://localhost:5304/" }, { "key": "id", "value": "#sep1" }, { "key": "type", "value": "SEP-TCS" } ] }, { "key": "testkey1", "value": "testvalue1" }, { "key": "testkey2", "attributes": [ [ { "key": "publicKey", "value": "1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ" }, { "key": "id", "value": "#pubkey1" }, { "key": "type", "value": "AUTHN-KEY" } ], [ { "key": "publicKey", "value": "1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ" }, { "key": "id", "value": "#pubkey2" }, { "key": "type", "value": "AUTHN-KEY" } ] ] } ] }, "Envelope": { "kind": "UDIDDocument", "encryptionFlag": "NotEncrypted", "hashedThumbprint64": "MpUTVq+AYTMEucWUFfMWfsWJRQ6tmO6FGzjAJGMN4T0=", "signedHashSignature64": "CLFgZCLJPzozxwB+JjJr7xQdZxgcwbEX4XBsujD+1rCW0sd6T4JFMVFTb86H50HQZ6h7myUld+9pIlbNWS3IPIg11uwYjlzMe32AO+ETCMSEJQJAPN9IJB//C4J2SkAdkK9OszStVsA/GYYtKZQdYSTdDESQCDVw6292N92bIJY=", "comments": [ "Bob's UDID Document", "It works!", "Created by TDW.TCSServer at 2021-06-15 07:07:09Z" ] } }

Sunday, 13. June 2021

reb00ted

Colorado is now the 3rd US state with modern privacy legislation, with a twist

This past week, the Colorado legislature passed the “Colorado Privacy Act”, which makes Colorado the 3rd US state with modern privacy legislation, following the CCPA in California and the CDPA in Virginia. It will start going into effect about 24 months from now. Its provisions are similar to California’s and Virginia’s, except for this one, which goes substantially further: A CONTROLL

This past week, the Colorado legislature passed the “Colorado Privacy Act”, which makes Colorado the 3rd US state with modern privacy legislation, following the CCPA in California and the CDPA in Virginia. It will start going into effect about 24 months from now.

Its provisions are similar to California’s and Virginia’s, except for this one, which goes substantially further:

A CONTROLLER THAT PROCESSES PERSONAL DATA FOR PURPOSES OF TARGETED ADVERTISING OR THE SALE OF PERSONAL DATA SHALL ALLOW CONSUMERS TO EXERCISE THE RIGHT TO OPT OUT OF THE PROCESSING OF PERSONAL DATA CONCERNING THE CONSUMER FOR PURPOSES OF TARGETED ADVERTISING OR THE SALE OF PERSONAL DATA … BY CONTROLLERS THROUGH A USER-SELECTED UNIVERSAL OPT-OUT MECHANISM THAT MEETS THE TECHNICAL SPECIFICATIONS ESTABLISHED BY THE ATTORNEY GENERAL…

In other words, Do Not Track – or something very much like it – is back in Colorado, and ignoring the setting, like companies did widely when Do Not Track was created, is not an option any more. The technical details will need to be figured out between now and when this provision goes into effect, which two and a half years away. So plenty of time to get this right.

Progress! Which US state is next? The IAPP has a great chart showing the state of privacy legislation in states around the US.

Here is the full text of the Colorado bill.

Friday, 11. June 2021

Kerri Lemole

Open Badges as Verifiable Credentials

In the W3C VC-EDU call on June 7, 2021 we discussed Open Badges asserted as W3C Verifiable Credentials (VCs). This call began the public discussion of Open Badges as Native VCs (potentially as Open Badges 3.0) to inform the IMS Open Badges Working Group. Why are we discussing this? Why does it matter? How will it work? Feedback from folks in the community have suggested that it would be helpful to

In the W3C VC-EDU call on June 7, 2021 we discussed Open Badges asserted as W3C Verifiable Credentials (VCs). This call began the public discussion of Open Badges as Native VCs (potentially as Open Badges 3.0) to inform the IMS Open Badges Working Group. Why are we discussing this? Why does it matter? How will it work? Feedback from folks in the community have suggested that it would be helpful to answer these questions first from a conceptual standpoint. In a later post, we can outline what the structural changes could look like.

Open Badges are digital credentials that can recognize learning, achievements, and even memberships. They look like an image, but inside the image are metadata properties (it works much like how digital photos have metadata properties that explain the data of the photo, location, etc. so that when the photo is uploaded, applications understand the data). The metadata describes why the badge was issued, when, the recipient (typically an email address), the description and criteria of the achievement, and, critically, how this digital credential can be verified.

The verification of an Open Badge is dependent on the Issuer of the badge. It can be done in one of two ways:

Hosted Verification — One of the metadata properties of an Open Badges is the assertion id which is a URL that hosts the metadata in a .json file. If that URL can’t be found, then the badge is not verifiable Signed Verification — An Issuer can digitally sign the metadata of a badge. This uses cryptography to ensure that the badge data has not changed and that the issuer was the entity that issued the badge. The public key used to sign the badge must be traceable back to the issuer.

Primarily, Open Badges platforms are issuing hosted badges. This means the verifying party is dependent on the issuer to host the data. Also, the issuer has some ability to track when the badge data has been accessed and potentially by who (or at least by IP address).

For 99% of the badges that have been issued to date, this is fine. In fact, badges are often shared via web pages that attractively display the image and data (not by the .json files that contain the data or the baked image with the data inside). So while Open Badges are both human readable and machine readable, the human readable approach is what is used most often. Signed badges are closer to Verifiable Credentials because they also rely on cryptographic proof.

Neither of these approaches give learners control of their data and this is the overall conceptual shift between Open Badges 2.0 and Open Badges as Verifiable Credentials.

Verifiable Credentials, like signed Open Badges, are signed by the issuer and verified cryptographically. Decentralized Identifiers (DIDs) can be used to identify the issuer and recipient (using DIDs in Open Badges 2.0 has been prototyped). Also, Verifiable Credentials have a concept called “Presentations” which can be digitally signed by the recipient to prove that they are the recipient of the verifiable credential(s) being presented. All in all, this means that the issuer is verified and also the recipient.

Verifiable Credentials can be displayed, shared, and presented from wallet applications that are web, mobile, or desktop based (most are mobile right now). Recipients can present one or more of their credentials to verifying third-parties who can verify the credentials without depending on the issuers and consume the data. Not only can the participant manage which credentials are being verified, they can control what aspects of their data are being shared.

For example, a student’s digital wallet may contain a digital passport, an Open Badge representing a completed course, and an Open Badge representing a student ID (all as Verifiable Credentials). A relying third-party, such as a potential employer seeking to fill an internship role, may need to verify that a student is over 18, has completed a course on communication, and is a current student. The employer’s website application can ask the student to provide this discrete information using their wallet and the student can do this without revealing any of the other information in those credentials like their date of birth, address, photo, or even the name of the school they attend. And all of this information can be verified without contacting the issuers of those credentials. Open Badges 2.0 can’t do this.

Verifiable Credentials put learners in the center of a trust triangle with issuers and verifiers. They also add an additional layer of verification for the recipients. Open Badges can take advantage of this, be the first education-focused digital credential spec to promote personal protection of and access to data, and be part of the growing ecosystem that is exchanging Verifiable Credentials.

It’s worth noting that the human readable aspect of Open Badges would not change in a VC version. Issuers can still display web pages for the issued badges and allow recipients to share their badges online from those pages. The difference being that those web pages would not be reliant on for machine verification or consumption.

Join us for this continuing discussion in the next VC-EDU call this coming Monday June 14 (and most Mondays) at 8am PDT / 11am EDT / 4pm BST / 5pm CEST. This call is public and all are welcome to join. For this Monday’s call (6/17), the zoom info has changed:

https://us02web.zoom.us/j/89426035740?pwd=dmQ0QS9wMUdFaGo0eFcwbkplV3RjUT09

Thursday, 10. June 2021

Bill Wendel's Real Estate Cafe

Peak real estate dysfunction & the K-shaped housing emergency, where do you stand?

Nearly four months ago, economist John Wake (who’s joined one of our Friday RECALL Roundtables), warned that the nation was facing a pandemic-induced, National Housing Emergency. … The post Peak real estate dysfunction & the K-shaped housing emergency, where do you stand? first appeared on Real Estate Cafe.

Nearly four months ago, economist John Wake (who’s joined one of our Friday RECALL Roundtables), warned that the nation was facing a pandemic-induced, National Housing Emergency. …

The post Peak real estate dysfunction & the K-shaped housing emergency, where do you stand? first appeared on Real Estate Cafe.


Damien Bod

Integration Testing for ASP.NET Core using EF Core Cosmos with XUnit and Azure DevOps

This article shows how integration tests could be implemented for an ASP.NET Core application which uses EF Core and Azure Cosmos. The database tests can be run locally or in an Azure DevOps build using the Azure Cosmos emulator. XUnit is used to implement the tests. Code: https://github.com/damienbod/AspNetCoreEfCoreCosmosTesting EF Core is used to the access […]

This article shows how integration tests could be implemented for an ASP.NET Core application which uses EF Core and Azure Cosmos. The database tests can be run locally or in an Azure DevOps build using the Azure Cosmos emulator. XUnit is used to implement the tests.

Code: https://github.com/damienbod/AspNetCoreEfCoreCosmosTesting

EF Core is used to the access Azure Cosmos database. An EF Core DbContext was created to access Cosmos. This is like any EF Core context, with the DBSet definitions as required. Some Cosmos specific definitions are added using the OnModelCreating method. See the Cosmos-specific model customization for more details.

public class CosmosContext : DbContext { public CosmosContext(DbContextOptions<CosmosContext> options) : base(options) { } public DbSet<MyData> MyData { get; set; } protected override void OnModelCreating(ModelBuilder modelBuilder) { modelBuilder.HasDefaultContainer("MyDataStore"); modelBuilder.Entity<MyData>() .ToContainer("MyDataItems"); modelBuilder.Entity<MyData>() .HasPartitionKey(o => o.PartitionKey); modelBuilder.Entity<MyData>() .Property(d => d.ETag) .IsETagConcurrency(); } }

The MyData class is is used to model the Cosmos documents. This has a PartitionKey and also an ETag which can be used for the Optimistic concurrency validation.

public class MyData { public string Id { get; set; } public string PartitionKey { get; set; } public string Name { get; set; } public string Description { get; set; } public string ETag { get; set; } }

The MyDataService service class is used to access the context and implement some query logic as required. I like to keep this simple and not separate the specification of the queries from the the business or the Linq statements. This reduces the amount of code and keeps the data access, business simple and makes it easy to adapt.

public class MyDataService { private CosmosContext _cosmosContext; public MyDataService(CosmosContext cosmosContext) { _cosmosContext = cosmosContext; } public void EnsureCreated() { _cosmosContext.Database.EnsureCreated(); } public async Task CreateAsync(MyData myData) { await _cosmosContext.MyData.AddAsync(myData); await _cosmosContext.SaveChangesAsync(false); } public async Task<MyData> Get(string id) { return await _cosmosContext.MyData.FirstAsync(d => d.Id == id); } public async Task<IList<MyData>> NameContains(string name) { return await _cosmosContext.MyData .Where(d => d.Name.Contains(name)).ToListAsync(); } }

The ConfigureServices method adds the services required to use EF Core and Cosmos DB. The services are used in a Razor page application, but this could be any web application, ASP.NET Core API or ASP.NET Core Blazor.

public void ConfigureServices(IServiceCollection services) { services.AddDbContext<CosmosContext>(options => { options.UseCosmos( "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==", databaseName: "MyDataDb" ); }); services.AddScoped<MyDataService>(); services.AddRazorPages(); }

The service needs to be tested. Instead of mocking away the database or using separate specifications classes as parameters, the service can be tested as one using Azure Cosmos emulator and EF Core. We used the framework tools to test our code. An EF Core in-memory database could also be used instead of the Azure Cosmos emulator. We use the emulator for these tests.

The tests are setup to add the services to the IoC and build these. The code can be run and asserted as required. To start locally in dev, the Azure Cosmos emulator needs to be started first.

using AspNetCoreCosmos.DataAccess; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using System; using System.Threading.Tasks; using Xunit; namespace AspNetCoreCosmos.DbTests { public class MyDataTests : IAsyncLifetime { private ServiceProvider _serviceProvider; public ServiceProvider ServiceProvider { get; set; } [Fact] public async Task MyDataCreateAsync() { using (var scope = _serviceProvider.CreateScope()) { // Arrange var myData = new MyData { Id = Guid.NewGuid().ToString(), PartitionKey = "Test", Name = "testData", Description = "test description" }; var myDataService = scope.ServiceProvider.GetService<MyDataService>(); myDataService.EnsureCreated(); // Act await myDataService.CreateAsync(myData); var first = await myDataService.Get(myData.Id); // Arrange Assert.Equal(myData.Id, first.Id); } } public Task InitializeAsync() { var serviceCollection = new ServiceCollection(); serviceCollection.AddDbContext<CosmosContext>(options => { options.UseCosmos( "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==", databaseName: "MyDataDb" ); }); serviceCollection.AddScoped<MyDataService>(); _serviceProvider = serviceCollection.BuildServiceProvider(); return Task.CompletedTask; } public Task DisposeAsync() { return Task.CompletedTask; } } }

The integration tests can be run in the Azure DevOps CI. I used a yaml file to test this and added this to my Azure DevOps build. This was a little bit tricky to setup because I did not easily find any working docs. The Microsoft.Azure.CosmosDB.Emulator is installed and started using Powershell. Then the tests can be run.

Note: Cosmos db emulator is pre-installed on the windows-latest vm hosted image on Azure DevOps . Docs here – https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md.


Thank you Lohith for researching this!

trigger: - main variables: solution: '**/*.sln' buildPlatform: 'Any CPU' buildConfiguration: 'Release' vmImage: 'windows-latest' stages: - stage: Build displayName: Build .NET sln pool: vmImage: $(vmImage) jobs: - job: Build displayName: Build pool: vmImage: $(vmImage) steps: - task: NuGetToolInstaller@1 - task: NuGetCommand@2 inputs: restoreSolution: '$(solution)' - task: VSBuild@1 inputs: solution: '$(solution)' msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"' platform: '$(buildPlatform)' configuration: '$(buildConfiguration)' - task: VSTest@2 inputs: platform: '$(buildPlatform)' configuration: '$(buildConfiguration)' - stage: IntegrationTests displayName: Integration Tests dependsOn: Build pool: vmImage: $(vmImage) jobs: - job: intgrationtests displayName: Run integration tests steps: - task: DotNetCoreCLI@2 displayName: Restore inputs: command: 'restore' - task: PowerShell@2 displayName: 'Starting Cosmos Emulator' inputs: targetType: 'inline' workingDirectory: $(Pipeline.Workspace) script: | Write-Host "Starting CosmosDB Emulator" Import-Module "C:/Program Files/Azure Cosmos DB Emulator/PSModules/Microsoft.Azure.CosmosDB.Emulator" Start-CosmosDbEmulator - task: DotNetCoreCLI@2 displayName: "Cosmos Database Tests" inputs: command: test projects: "**/*.DbTests.csproj"

You can add the yaml pipeline to your Azure DevOps build and it will run like the triggers are defined or the Azure DevOps policies.

This works good, but you have to be careful in preparing the tests and running in parallel. Implementing the tests like this means you have less code in your application and you can still fully test all your code. A disadvantage with this approach is that the tests take longer to run compared to unit tests without the emulator.

Links

https://docs.microsoft.com/en-us/ef/core/providers/cosmos/

https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests

https://docs.microsoft.com/en-us/azure/cosmos-db/

https://dev.azure.com/

https://xunit.net/

https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator

Wednesday, 09. June 2021

Matt Flynn: InfoSec | IAM

Bell Labs, the Colonial Pipeline and Multi-Factor Authentication (MFA)

A simple technology invented by Bell Labs over 20 years ago (and widely used today) could have prevented the Colonial Pipeline attack. In 1880, the French government awarded Alexander Graham Bell roughly the equivalent of $300K as a prize for inventing the telephone. He used the award to fund the research laboratory that became colloquially known as Bell Labs. If you’re not familiar with Bell

A simple technology invented by Bell Labs over 20 years ago (and widely used today) could have prevented the Colonial Pipeline attack.

In 1880, the French government awarded Alexander Graham Bell roughly the equivalent of $300K as a prize for inventing the telephone. He used the award to fund the research laboratory that became colloquially known as Bell Labs. If you’re not familiar with Bell Labs, you should be. In the 140+ years that followed, researchers at Bell Labsinvented radio astronomy, transistors, lasers, solar cells, information theory, and UNIX, just to name a few of the many accomplishments. Among the many prestigious awardsgranted to Bell Labs researchers are nine Nobel prizes and twenty-two IEEE Medals of Honor.

In 1998, I joined AT&T Labs, which was a research group that the company retained when they spun out most of Bell Labs to Lucent Technologies in 1996. I was a Web Application developer; one of the least technical roles in the Labs. If I ever thought for a moment that I knew technology, I was quickly humbled when I built an app that tracked the Labs' actually importantprojects. The experience of working in the Labs stuck with me in the form of humility and curiosity. I accepted that I may never be the foremost expert in any given technology and I assumed the mindset of a forever student. Even today, I constantly question what I think I know because there are always holes in my knowledge or perspectives that I haven’t seen.

1998 was the same year that researchers at AT&T Labs were issued a patent (filed in 1995) for what became known in our industry as Multi-Factor Authentication (MFA). As a Product Manager at a tech firm, I don’t review patents for legal reasons. But I recently saw an excerpt of the abstract for the AT&T patent and there was one line that I found entertaining: “A preferred method of alerting the customer and receiving a confirmation to authorize the transaction back from the customer is illustratively afforded by conventional two-way pagers.” Not much has changed in 23 years. Pagers have been largely replaced by SMS but text messaging through the telecom provider’s network remains one of the most popular delivery mechanisms for MFA (despite some potential security flaws). 

I have no personal insight into AT&T’s motivations at the time, but I read Kevin Mitnick’s book a few years ago (Ghost in the Wires) and can’t help but wonder if AT&T was at the forefront of developing security technologies because they were such a target of hackers for so many years. I also reached out to Steve Greenspan, one of the inventors named in the patent to get his thoughts on the project. He noted:

"Two-way pagers had just come out (1994-1995), and our cybersecurity friends were debating whether quantum computing would undermine password-based security. The goal was to explore business applications for two-way pagers and to put humans in-the-loop for secure access."

Quantum computing is a a pretty interesting business driver for MFA, especially in the mid-1990's. The concern is even more relevant today as we inch closer to quantum compute becoming a practical reality. Today's authentication systems should store password data in non-reversible hashes (theoretically preventing the quantum threat), but it's clear that credentials are being stolen all the time (often via large databases that are just left unprotected) and MFA remains a top solution to mitigate the damage. Steve and team were clearly on the right track when they dreamed up out-of-band authentication and deserve some credit and recognition for the foresight.

You may be wondering how this relates to the pipeline attack that led to fuel shortages across the U.S. East Coast. Bloomberg reportedthat the Colonial Pipeline, which is the largest fuel pipeline in the country, was taken down by a single compromised password. That should never happen given the variety of tools available to limit and control access, starting with MFA – a relatively simple solution that would likely have prevented the attack. The entry point to the system was a Virtual Private Network (VPN) account. If you’re using a VPN and expose anything sensitive inside the VPN, you should implement strong authentication that includes at least two authentication factors (something you know, something you have, something you are). These are widely available technologies that are very effective against lost or stolen credentials.

Of course, authentication isn’t the end of the story. Today’s widely distributed and highly dynamic environments require multiple layers of security. We all know how popular email and phishing attacks have become. It only takes one person inside a network to open an email, click a link, or logon to a phishing site to give an adversary a foothold in the network. We have to assume that will happen and build layers of strong security between any one user and the potential targets.

To illustrate the point, here’s a quick example:

Grocery stores who sell small, high-value items have traditionally struggled with theft. (Ask me over a beer sometime about how I helped take down a recurring thief when I worked at a grocery store.) If the only answer was to authenticate users (check ID) on the way into the store, it wouldn't be enough. Once inside, someone can still pocket items and walk out without paying. If you walk into a grocery store today, you’ll see cameras in the healthcare aisle where small, expensive medications line the shelves. But that’s not enough either. Each item is also locked in an anti-theft device that’s removed at the register. And some items are found in a locked cabinet that requires employee assistance. Theft still happens, but each layer reduces the risk. Our IT environments are much more complicated in terms of the various pathways to theft and our responses to reduce risk typically require more than a few layers of security.

Sensitive data should only be stored in a secure area of the network with access controls and Least Privilege enforcement. Access should be limited to specific hosts or networks. Data should be encrypted (inside the file when possible - so if the file is stolen, the data is still unusable). There should be strong authentication to get into the network and monitoring of all activity. There should be alerts on unusual behavior and Data Loss Prevention (DLP) to evaluate the sensitivity of data moving across the network. The environment should be scanned regularly for vulnerabilities and misconfigurations. And on and on. Any one of these security mechanisms alone is not enough. This multi-layered approach to security is critical in developing a strong security posture that minimizes risk.

We could argue about where to start or which security controls are most important. But, it seems like a no-brainer to implement MFA for employees accessing corporate data and applications. Microsoft, who deals with 300 million fraudulent sign-in attempts daily concludedthat “MFA can block over 99.9 percent of account compromise attacks.”That sounds about right. While targeted attacks have increased in prevalence, most attacks are not targeted at specific companies or individuals. Most start with automated scripting or broad-scale phishing attacks that span across potentially thousands of companies and/or millions of people at the same time. When a foothold is found (a script finds a vulnerability or an open port, a user enters credentials into the phishing site, etc.), the attack begins. Implementing a few simple security technologies like automated vulnerability scanning and MFA can prevent most attacks before they begin. Even if a sophisticated phishing attack succeeds despite MFA, the credentials will not be very useful beyond the initial session (which should be limited in scope by other controls).

No single technology will solve all cybersecurity problems. But, implementing MFA is low-cost, easy-to-implement, and highly effective. It may even make life easier for end-users. Password requirements can be loosened because there’s less risk associated with cracked passwords. And there are numerous implementations of passwordless authentication that, while they may not always meet the strict definition of MFA, provide similar (sometimes higher) levels of security as MFA without requiring a password. Combined with context-aware adaptive security (that verifies device, network, location, time-of-day, etc.), these passwordless authentication options may provide the right level of balance between security and user experience. At this point, this isn’t scare tactics or FUD. Attacks on National infrastructure or other high-profile targets can impact the lives of millions with a single execute command. MFA is an easy layer to add to improve security and it’s commonly included with authentication solutions, so there’s really no excuse. It’s time to get it done.


Werdmüller on Medium

Ma.

We lost my mother, Deborah Monas, on Sunday evening. I was giving her a head rub; my sister Hannah held her hands; my dad Oscar and her brother Steve were at her feet. How is this possible? The world doesn’t feel real. When I was a small child, we used to pull out the sofabed in our postgraduate student flat and watch Doctor Who together under a blanket. What I would give for a TARDIS

We lost my mother, Deborah Monas, on Sunday evening. I was giving her a head rub; my sister Hannah held her hands; my dad Oscar and her brother Steve were at her feet.

How is this possible? The world doesn’t feel real.

When I was a small child, we used to pull out the sofabed in our postgraduate student flat and watch Doctor Who together under a blanket. What I would give for a TARDIS now: a way to correct the fabric of time and space. Clearly something is badly wrong with it.

I want to have something perfect to say, some beautiful encapsulation of who she was, but the truth is that nothing could be perfect enough. There’s no way to write a biography that isn’t an imperfect model. Nothing can adequately describe my beautiful mother: her overwhelming kindness, her sense of justice, her irreverence for tradition and institutions, the love that our house was always, always filled with.

I’ve written a lot about her health in this space. She had dyskeratosis congenita, which first expressed itself as pulmonary fibrosis. She fought it hard. When she began to use supplementary oxygen, my sister and I moved to California to be closer. The double lung transplant she received at UCSF gave us almost an extra decade of time with her, and it was a privilege to be by her side, with my dad and my sister, on this journey.

She loved reading, and both Hannah and I got to share those experiences with her. Hannah read to Ma for years: books by authors like Wade Davis, Octavia Butler, and Tommy Orange. Ma and I shared book recommendations; this year we read Caste and The Nickel Boys, among others. Even as it became harder to read in print, she picked up audiobooks, and kept going, recording what she’d read in a notebook she kept by her bed.

Years ago, we were gifted a Sunfish sailing boat by our family friend Pammy Biscoe. The two of them had sailed on it when they were much younger. Now it became her and my thing: she was happiest out on Waquoit Bay, our striped sails catching the wind just right. She called it a Whoosh. I’ll remember her in an infinity of ways, in an infinity of moments, but being out on the water with her, watching her smile as we picked up speed, is one I’ll treasure the most.

When my parents met, in Berkeley in the 1970s, she worked to fight for affirmative action and tenants’ rights. Her belief and support for progressive causes was an anchor throughout her life. We discovered this week that she had quietly made over 178 progressive donations last year alone, without any of us really knowing. A list of some of the causes she consistently supported follows at the end of this post; in lieu of flowers, we’re asking people to contribute if they have the means.

I want to honor her by furthering what she put into the world. The loving, non-conformist, irreverent, equity-minded spirit that she embodied.

As she lay in her hospital bed, we read Ma messages from people who loved her throughout her life. One, from Hannah’s friend Anita Hurrell, particularly sums up our collective childhood, and the sensibility I want to take forward in my own life, inspired by my mother. I’ll end by sharing it here, with kind permission.

Dearest Deb,

One time you drove us in the van to the seaside and we ate sandwiches with cucumber in them and I thought they tasted delicious and I felt this strong sense of deep content sitting with Hannah in the back listening to her singing and humming for the whole journey. I have no idea where we went, and in my head it was nowhere in England, but rather part of the big-hearted, loving, funny, relaxed, non-conformist world of your family in my childhood — full of your laughter and your enormous kindness. Sitting on the bench in your house in Marston I recall a moment of feeling complete certainty that your pirozhki were the yummiest food that had ever been made. Staying in Wheatley when my mum and dad had gone away we ate popcorn and I felt safe even though at the time I had lost Blue Bear. I remember calling you my second mummy. I’m not sure I was always a very nice kid and was probably very ungrateful then, but now I wish I could convey how I revere you, Oscar, Ben and Hannah in my thoughts and how lucky I feel to keep your example with me. I look back and see how talented Hannah already was then when we were so little and she could just sing and sing and draw in a magic way, how cool she was (her doll Manuela had beautiful black wild curly hair while I chose Charlotte with twee clothes and ridiculous ringlets), what a true feminist you were, how much of parenting you seemed to do much better than we do these days, how generous and homemade and fun and kind the world you and Oscar made was.

You are an asset to the universe. I will always love you very much.

Anita

Causes Ma consistently supported:

ACLU — donate

Planned Parenthood — donate

Progressive Turnout Project — donate

NARAL Pro-Choice America — donate

314 Action Fund — donate

Stop Republicans — donate

National Democratic Training Committee — donate

End Citizens United — donate

KQED Public Media — donate

The Squad Victory Fund — donate

BOLD Democrats PAC — donate

National Bail Out — donate

Phil Arballo for Congress (CA-22) — donate

Scott Sifton for Senate (MO) — donate

Originally published at https://werd.io on June 9, 2021.

Tuesday, 08. June 2021

Timothy Ruff

The EU Announcement is the Biggest Ever in SSI

On June 3, 2021 a significant thing happened for SSI (self-sovereign identity): The European Union announced a plan for an EU-wide digital wallet that would support drivers licenses, tax filings, opening bank accounts, prescriptions, academic achievements, seamless travel experiences and proving one’s age to enter a nightclub, among many other possibilities.* It’s easily the biggest thing to ever

On June 3, 2021 a significant thing happened for SSI (self-sovereign identity): The European Union announced a plan for an EU-wide digital wallet that would support drivers licenses, tax filings, opening bank accounts, prescriptions, academic achievements, seamless travel experiences and proving one’s age to enter a nightclub, among many other possibilities.*

It’s easily the biggest thing to ever happen to SSI, and arguably the biggest to ever happen to identity generally, digital or otherwise.

Read more at my new blog location, here: https://credentialmaster.com/the-eu-announcement-is-the-biggest-ever-in-ssi/


MyDigitalFootprint

Day 0 CDO language. The translator, interpreter and go-between

Whilst our ongoing agile iteration into information beings is never-ending, there are the first 100 days. But what to focus on? Well, that rose-tinted period of conflicting optimisation is what </Hello, CDO!> is all about. Maintaining sanity when all else has been lost to untested data assumptions is a different problem entirely. On Day zero of being a #CDO, you have to be ready and pre

Whilst our ongoing agile iteration into information beings is never-ending, there are the first 100 days. But what to focus on? Well, that rose-tinted period of conflicting optimisation is what </Hello, CDO!> is all about. Maintaining sanity when all else has been lost to untested data assumptions is a different problem entirely.

On Day zero of being a #CDO, you have to be ready and prepared as a translator, interpreter and go-between. Yes, the essential “translation” of business needs into information requires identifying the appropriate data, the relevant analysis, and the correct interpretations, but that is not what I am talking about. There is a different translation to the appropriately modelled, described and analysed, data that offers the language to enable siloed departments in organisations to talk to each other.

The CDO must have translation skills to help other executives talk about what data means to them and that each party leaves with a common understanding. Exceptional executives can ensure that key concepts from one domain or department are appropriately translated into language and concepts in the other domain.

By example,

Data in the language of the finance department has to do with how to position data revenue and costs in the activities in the finance team use. Getting the budget spent on data, algorithms and analysis into the proper R&D tax claim, the capitalisation table and the correct allocation for depreciation take times. How and when should data revenue be recognised? How should the margin for data be calculated? It has nothing to do with what the data does. Understanding what KPI’s the CFO is tasked with and how you can help them be achieved makes for an excellent working relationship. If The CFO is tasked with improving the balance sheet, how you capitalise spending is critical. Data in the language of the legal department is a focus on the contractual and regulatory obligations. Understanding what the General Council is responsible for is critical, as it has nothing to do with the value data creates for the organisation. If you want certain protections for data, data use and data sharing, working with the legal team to craft holistic on and offline terms that spread across all contracts is critical. Data in the language of the marketing team can be about privacy persevering data sharing and trust, leading to additional brand value. Sitting down with the senior leadership team in marketing to understand their KPI and BSC can help you translate data into what they need to achieve their goals. It has nothing to do with the value of the data; how they use data is a different issue. However, the CDO must be on top of conflicts that arise with sales targets driven by incentives, KPI’s opt-in demands and contractual boundaries.

Being the translator, interpreter and go-between form you like a bridge, but as already articulated, it means you are also the new corporate punchbag.

Note to the CEO

The CDO must have translation skill to help your executives speak with other disciplines about data and enable each party to leave with a common understanding. All CDOs will find the language to translate business context into data-enabled initiatives and ensure that key concepts from one domain or department are appropriately translated into language and concepts in the other domain.

We focus on the skills to “translate business needs” into data and information needs during interviews and the recruitment cycle. This additional translator, interpreter and go-between skill in this article are the built-in skills for exceptional CDO’s. However, if there is someone already in the team with this strength, it is not essential, but it does create a better working culture. Should you need these skills, they must be proven to you before you offer the job, it is not one to learn on the job.

Monday, 07. June 2021

Phil Windley's Technometria

Building an SSI Ecosystem: MemberPass and Credit Unions

Summary: How does a functioning credential ecosystem get started? This post looks at MemberPass, the credit union credential ecosystem and explores the strategies credit unions are using to gain broader adoption. My work in self-sovereign identity began with credit unions. It was March of 2016 and I was having a conversation with Timothy Ruff and Jason Law of Evernym about how difficult i

Summary: How does a functioning credential ecosystem get started? This post looks at MemberPass, the credit union credential ecosystem and explores the strategies credit unions are using to gain broader adoption.

My work in self-sovereign identity began with credit unions. It was March of 2016 and I was having a conversation with Timothy Ruff and Jason Law of Evernym about how difficult it would be to get a multi-sided verifiable credential market going. Timothy's response was "You've got to come to Denver next week!" I showed up at a hotel ballroom in Denver to find almost 100 executives from credit unions all across the US clamoring (no, really) for verifiable credentials. I was hooked.

Over five years later, with a few fits and starts, credit unions are deploying credential-based identification systems for their members. To date, seven credit unions have issued credentials to over 22,000 members or about 2% of the eligible membership of those same credit unions.

Why do credit unions care? One word: fraud. Or maybe two: fraud reduction.

It's All About Authentication

Credit unions and their members face the threat of fraud on all sides. And credit unions employ lots of tools to fight it. But ultimately, the problem comes down to the member and credit union authenticating each other. The problem is that doing this securely annoys people.

None of us like to spend a minute–or more–answering security questions at the start of a customer service call. And SMS-based multi-factor authentication is becoming increasingly fraught. Is that text you just got warning you about fraudulent charges on your credit card really from the credit union? It's hard to tell.

Early on, a few intrepid people in the credit union industry recognized that self-sovereign identity (SSI) offered a way out of this mess. Credit unions are often small and band together to form credit union service organizations (CUSOs) that provide them the services they can't build on their own. They formed a CUSO called CULedger (later renamed Bonifii) to make that vision a reality. Bonifii offers an SSI-based solution for credit unions called MemberPass.

MemberPass Trust Triangle (click to enlarge)

MemberPass allows credit unions to offer their members a verifiable credential that they can use to prove their member number to the credit union. Initially, the MemberPass credential schema is fairly simple, containing only the following attributes:

CredentialDescription CredentialId MemberSince MemberNumber CredentialName Institution

Of course, credentials could be much more complicated than this, but this simple schema is sufficient for a member to prove they are in possession of a credential for a specific member number. Members use the MemberPass wallet to connect to the credit union and hold the MemberPass credential.

MemberPass relies on Bonifii's partner Evernym for technical services. Credit unions integrate their back office applications with the MemberPass platform at Bonifii which relies on cloud services provided by Evernym.

MemberPass Architecture (click to enlarge) Growing Adoption

While much of the response to fraud is reactive, MemberPass is proactive. Credit unions work to get members using MemberPass as an active measure to prevent fraud. As I said earlier, to date, seven credit unions have issued credentials to over 22,000 members or about 2% of the eligible membership of those same credit unions. Julie Esser, Bonifii's SVP of Client Engagement expects the number of credit unions using MemberPass to more than double in 2021 and the number of eligible members to jump by almost an order of magnitude.

Increasing the number of credit unions using MemberPass is the first segment in the adoption journey. MemberPass is already integrated with some of the back office platforms that credit unions use, easing the journey. Bonifii is also working with third party integrators to ensure they're technically ready to do the integrations for the rest.

The second segment of the adoption journey is increasing the percentage of members enrolled from the current 2% to 5% and then 10% over the next year. To do that, Bonifii works with credit unions to train frontline staff in the enrollment process. Early enrollments are happening in the branch. But enrollment can also happen on the phone. The phone enrollment process takes 3-5 minutes. The member receives the MemberPass credential while they're on the phone so the call center agent can help with any problems.

First Education Credit Union's President, Jim Yates, says that most new members are signing up. Signing up the larger body of existing members will likely require a move to self-enrollment since many never come into a branch. Self-enrollment is possible within the authenticated context of the credit union's web site. If the member chooses to enroll, they'll be directed to download the MemberPass app and then scan a QR code. This establishes a secure DIDComm connection. The credit union can then make the MemberPass credential offer. UNIFY Financial Credit Union allows self-enrollment now their online banking application.

Once a member is enrolled, the credential can be used in-person at the branch, in the drive-thru lane (with or without interactive teller machines), on the phone, or online. This is not only more secure, but often more convenient as well. For example, someone going through the drive-thru lane can authenticate without passing plastic credentials back and forth. Logging in no longer involves receiving a text and then typing in the code. And calling into the call center no longer requires answering a series of questions of questionable value.

Instead, a push notification on the member's phone asks them to verify they're the one transacting with the teller, call-center employee, or web site. The member clicks "accept" and they're done. Behind the scenes, this is a proof request made through the already established DID connection. By clicking "accept", the member is responding to the request and proving attributes from their MemberPass verifiable credential.

And it's a win for the credit unions too. Desert Financial's EVP Ron Amstutz says it's an important step in reducing fraud. Desert Financial knows they're talking to a member and the member knows they're talking to Desert Financial. Desert Financial is initially recruiting members for the program who call into the call center frequently since that's a big pain point.

Zach Eychaner from 4Front Credit Union says the call center is the first focus for them as well. They are able to shave 30-40 seconds off of each call. With 20,000 calls a year, that time adds up.

The Road Ahead

The MemberPass credential with its limited set of attributes is just a start. The future could include using MemberPass at an ATM or to open account at another credit union. Bonifii's Esser says "Once they get used to MemberPass, members will expect to use it everywhere."

Here are a few things that credit unions could do to make more use of credentials and SSI:

As we've seen, the current MemberPass schema is very simple–it doesn't even include the members name. A schema with more information in it–information that's been validated by the credit union–would make it usable outside the narrow use case of authenticating the member to the credit union and offer more value to members. Credit unions could offer a pre-approval credential for loans that the member could hold and then use when they were ready for a loan. Bonifii could issue a credential for KYC use at credit unions, banks, and in other financial transactions. Shared branching is a hot topic in the credit union industry right now. Twenty-three thousand branches looks like a mega bank. But the identity fraud problems are even harder to solve across credit unions. MemberPass can help make shared branching a reality. Employers and employee groups historically make up the foundation of credit unions. Credit unions could partner with employers to create a credential ecosystem. The DIDComm connection is a secure messaging system. Credit unions can use this secure channel for sending notifications to members, or for customer service.

The lessons from MemberPass and the credit union industry are important for anyone launching a credential effort:

Pay attention to the process and tailor it to your industry. Fraud reduction is the focus. Credit unions are evolving their enrollment process and targeting the parts of the process where they can get the most leverage. Start simple. MemberPass is a simple credential but it serves an important purpose: reliably authenticating the member to reduce fraud. Plan for the future, but don't get distracted. There are a thousand use cases for credentials in financial services. Get some early wins with your simple "MVVC", minimum viable verifiable credential, before you move on to the rest. Stay the course. Building a credential ecosystem is more about human factors than technology. In the words of Julie Esser "The technology is baked." But that's just the start. The MemberPass ecosystem is complicated by regulation, scale, and a decentralized collection of players, each with their own problems and goals. Building an ecosystem in this environment isn't easy, but it's where the reward is.

The Covid-19 pandemic caused credit union branches to close and call center volume skyrocketed and drive-thru lanes were crowded. As a result, fraud also increased. This created a heightened awareness of the importance of digital identity across the credit union industry. But while the pandemic might have pushed things along, many in the credit union industry had already concluded that self-sovereign identity was an answer that was not only flexible, interoperable, and secure, but also one that was aligned with the values of the member-owned cooperatives that make up the credit union industry.

Tags: identity ssi use+cases verifiable+credentials finance

Friday, 04. June 2021

MyDigitalFootprint

The shadowy hierarchy

I remain curious about how I can make better or wiser decisions.  I am sharing this as part of my journey as I unpack my own boundaries and models that prevent me from making better decisions.   Context I have personally, and will always, dislike and distrust “traditional” hierarchy, probably because I perceived that the “power” wielded on me would never be available to me.   I
I remain curious about how I can make better or wiser decisions.  I am sharing this as part of my journey as I unpack my own boundaries and models that prevent me from making better decisions.  
Context

I have personally, and will always, dislike and distrust “traditional” hierarchy, probably because I perceived that the “power” wielded on me would never be available to me.   I was always on the outside; it is the joy of neuro-diversity that you become aware at an early age that to fit in the system and structure, you have to align to it, which for me, had no natural alignment.  You either fight to fit in, fight the system or create your own.  For many fitting in is natural, for me it never happened, and I stupidly opted for creating my own.  I rebelled against the system and structures as I could only see hierarchy as a method of control to something I did not align to - telling me to do things that made no sense.  Write with your right hand as a lefty.

I am not alone; from Machiavelli to Second lieutenant Boris Drubetskoy in Tolstoy's War and Peace, many have realised that there are two structures of power.  There is an obvious hierarchy of control, reporting and subordination, and there is a second unsaid and unwritten one of lobby and access.  The obvious one is easy to rebel.  The lobby and access inner ring is where true power rests; it is the one to try and join. However, it is invitation-only, and many will sacrifice more than they realise in terms of ethics, morals and beliefs to be at the table and remain part of it. When thinking about data, bias and decision making, the incentive to join and incentives when in the club is a critically important driver. This unsaid shadowy hierarchy creates outcomes we did not plan for as some will jeopardise judgment and trust to be rewarded.  I would not align to this one either, and it was self-evident that you can only become a partner in a big firm when you have a big mortgage, depending on a big salary. The significant debt and dependence on salary provide leverage.  The humour in “Yes Minister”, the 1980’s BBC  political satire sitcom, exposed much of this.  

Why does this matter?  We increasingly talk to the equality and diversity (race, gender and neuro) agendas at the board and in the senior leadership team, but it is increasingly evident that there is an inner circle, which means we are not making smarter or better decisions.  The targets for transparency and equality are just that, a target.  (says the white over 50 male!)

I appreciate that it is difficult to separate the two dependent and co-joined aspects of the hierarchy. One aspect represents power and control, which is realised in terms of budget, size, scope, authority, and political dominance. The second is how and where important decisions sit, but those who lobby may be outside of the insights that data now provides for complex decision making.   

The separate but parallel structures have worked for a long time as the consequences of long term decisions and intergenerational outcomes were not rewarded or measured. ESG, climate and the brutality of our assault on our environment, coupled with data, means this is all changing, but those who gave everything to join the inner circle have an incentive to maintain the status quo for as long as possible. 

What ghosts are in the system that means decisions we make, with the best intentions, do not create the outcomes we desire

Take a moment to reflect.  Are there separate hierarchies in your experience, or are they the same thing? It matters today more than ever before as we now have to make long term decisions against the backdrop of our systems of short term incentives and rewards.  The inner ring becomes blind to a structure that always gives the feel of power as their short term decisions are highly rewarded through immediate incentives, which means being in the club is valuable for the few.  

This viewpoint is looking at these two hierarchies (formal and informal) but positioning where either can be in the shadow of the other. Why am I writing this because we appear to find short term incentive-driven decisions easy but struggle to make long term judgment?  I do not doubt the integrity of leaders who want to make better long term decisions using data and be better ancestors but get frustrated that it does not work.  There are likely ghosts from the old decision making and lobby hierarchy in our current heuristics that create outcomes that were not planned. 

Note: The term “Shadow Hierarchy” is used in management textbooks to describe the difference between a formal published and public structure and the one that actually has power and decision making.  Googling “The Shadow of Hierarchy” will lead you to the 2008 paper from Adrianne Heuritier and Dirk Lehmkuhl, which was part of an EU funded project looking at “new models of governance.”  


Our past is not easy to face up to. 

There appears to be a much longer story that starts when decisions, power and control were all united.  Over the past 5000 years, we have become increasingly short term focused and separated the decision-making process for the long term from power and control based on the science of management incentive and short term goals. There is no doubt that the structure of economics, along with other biases, come to play.  However,  we have created data technology on top of complex relational dependencies that means signals that should become noise through the layers of analysis, in fact, become significant distracting “interruptions” at the board. Whilst in the old system, noise and signals were hidden by the informal lobby that prevented more thoughtful long term decision making based on data, today they (signals and noise) create paralysis at all levels in structures as everyone knows everything.

Should the head of a religious movement be leading planning for the next 100 years or focussing on if the budget is spent according to the plan this quarter?  Should our political leaders face the daily news headlines and respond or ensure we are equipped to face a global pandemic?   Should the head of state be allowed to focus on sustainability, climate and global concerns or defending their grandchildren’s choices? What makes noise and news dominate interruptions.  In a joined power and decision hierarchy structure a long time ago, a few individuals could make those decisions and choices, and lobby worked.  Today our hierarchies,  analogous to decision making,  have become paralysed, confused and ineffective as the volume of data and signals mixed with noise has risen to a level where our ability to know what is the right thing to do is broken. Currently, we have not transitioned to something that works. Indeed lobby has also failed as it has become increasingly linked to short term reward and incentives.

The figure below captures this concept on a two-axis chart of scale and impact.  The bottom right being an idea, it has no scale and little impact.  The top right is big government, global businesses, global NGO’s and charities, global religion and large scale movements.   On the journey from the idea to scale, we either transition from our ability to make long term decisions to focus on a quarter by quarter reporting justifying the delta between actual and plan *or* hold onto inter-generational consequences. Risk, funding, capital and markets have a significant impact on the loss of that long term thinking, as the rewards for the players become aligned to short term incentives. Whilst the long term hierarchies become corrupted, the shadow hierarchy of lobby gives way to a different incentive and power game.  Impact and scale create the same problems irrespective of the organisation; short-termism can be recognised and rewarded.   

Joseph, as in the Bible story and multicoloured dream coat fame aka the Tim Rice and Andrew Lloyd Webber musical. It is one of the earlier written examples of man’s capability for longer-term planning.  It was a 14-year cycle, so not long term; however, 7 years of abundance followed by 7 of famine. Grow, harvest, store and distribute later. 4,000 years on and a 14-year strategic planning cycle looks massive but still short compared to the famed 100 years plus China and Japanese plans. I may have rose-tinted glasses that we were once better at long-range forecasting, but this is a good piece from three world-leading experts “is humanity, in fact, unable to successfully plan for the long-term future?”   In the context of our currently limiting systems - yes.  We have to break the system. I smell revolution.

Complexity of relationships

No doubt, a small part of the issue with our inability to long-range plan is the management of the complexity in relationships.  The web of collaborative relationships that we need to consider only ever gets more strained, detailed, involved, dependant, and unbalanced, as each party align to their incentives and rewards.  Critically, the unbalanced nature of the relationship means it is increasingly difficult to predict outcomes and reactions. (search:  emergent complex systems). Our commercial framing is explored in the figure below.

The shareholder has a dependency on the board to make a decision that supports their funding of the capital. In contrast, the directors are accountable and responsible for their decisions, including unlimited liability. Everyone now has a voice that can affect choices and actions.  The Director/ regulator/ ecosystem axis is dominated by who has the accountability and responsibilities, which are different for each stakeholder in an ecosystem and are often driving in different directions because of Balanced ScoreCards (BSC), KPI’s, incentives and being in the club.  OKR (objective and key results) are no better, and the difference between built vs operate is a false one for the long term.  Building the wrong thing can be rewarded very well in OKR land. 

Continuing around the model, there is a remarkably fluid relationship between the executive teams’ and the board, where the board depends on the exec team. Still, the exec team is accountable to the board.  (The level of fluid varies by nation and company law.) The relationships between and with the customer are particularly misunderstood, but ultimately the law says the directors are held accountable. Each of the roles in the chain of relationships from shareholders to the executive team required individuals capable of dealing with complex judgement.  However, this assumes that the first part of this viewpoint is null.  Let’s expand. 

We face three connected issues but not mutually exclusive because they are a team or part of an ecosystem.  

Some individuals in positions of influence and power are part of the club or want to be in the club and therefore have incentives to be in the club and stay in the club.  Their judgement and actions are aligned with being in the club incentives.

Some individuals have reached influence and power but cannot grasp the new mental models and skills for complex judgment.  Their judgement and actions are aligned with their experience.

Some individuals can understand the complexity and seek to explain and justify decisions and actions. 

Taking this thinking and plotting this on skills/ ability vs decision environment.

The diagram above shows a decreasing number of skilled individuals who can cope with complexity. This is because the training system is about effectiveness and efficiency and not about determining if you are doing the right thing (efficacy).  Many arrive at senior roles and find they have to shift their mindsets; some do, and some don’t, but they both now have decision capacity. 

For example, we know that systems that create inequality, insecurity, and unsustainable practices are not easily transformed.  Think of our government and economy. We have a system where 95% of the world live meal by meal, day by day, week by week or month by month.  An additional 4.99% can survive for 6 to 8 months on the available cash and funds. Less than 0.01% (80 Million) of the world’s population can plan for more than a year.  When you have those few in power and lobby, will they ever need to vote for change?  Worth following and reading Umair Haque; I love this essay “How the Economy is Designed to Keep You Poor and Powerless.” 

The laws of 1% explore outcomes if we change everything by + or - 1% and what it means to human behaviours.

What do “ghosts in the system” look like? 

Imagine you are at the fairground, and there is one of the stalls where you get to throw something to win a prize.  In the UK, we have the Coconut Shy.   You pay to get three balls or bags, stand behind the line and throw them one at a time at your coconut of choice.  Knock a coconut off, and you win it.  A simple game of skill (apparently).  However, when there is a ghost in the system, it is not so simple. You line up your ball on coconut number one (it is the biggest one) and throw it with all your skill.  As the ball approaches, the coconut moves and your ball sails past. You line up ball 2, aiming again at the largest coconut (the biggest prize); this time, you miss, but coconut number 4 wobbles.  Your last chance on this budget.  Lining up coconut number one for the last time, you hit it, but it does not fall off, but coconut number 6 does, the smallest one.   The ghosts win.   Your decisions and outcome was coconut number 1; you got number 6.  It was not your lack of skill; coconut number 1 is glued on. Different motivations and rewards.  

The ghosts make signals and noise. 

This viewpoint started from hierarchy, and I want to return to thinking about the two hierarchies, one of decision and one of power and unpack the issues that the abundance of data has created for us.  The diagram below sits with the idea that there is a natural order of decision making and power.  The movement from the bottom to the peak is a move in the decision time horizon.  Overlaid on this model is VUCA (volatile, uncertain, complex and ambiguous) situations. VUCA came from preparing/ training the solder who would have to face a situation where they had to decide. 

The hierarchy is worth a paragraph to unpack, as the context of VUCA is situational.  Front line workers are trained to be reactive. They know how to act based not on scenario but based on the situation (police, army, emergency, fire, medical, call centre, customer-facing representatives).  How they react is one of the rules, heuristics and repeated numerous times.  As the NIKE brand says - “just do it”, which is what you can do when you have trained to do the same thing for 10,000 hours - you don’t have to think it is a reaction.  Above this line is the management who ensure the environment the workers have has the lowest possible risk.  Above them is management, who think about “how do we adapt to new situations and threats?” Above them is the strategy layer, which considers “what skills do we need for the next five years, where are the gaps, and how do we access the skills.”   For most companies, this takes us to the CEO.  However, in public operations, there are two more layers. The transformational one is thinking about the infrastructure for the next twenty years and, finally, the policymakers.  The policy leadership should be thinking 50 years hence and considering what policy we will need to form and how. Even at this simple layering, we can see that global leaders from presidents, prime ministers, and heads of state struggle to plan for 5 years yet are tasked with 50.  We are not 10 x better; we have created a system 10 times worse. 

What we should be witnessing is that one layers signal is another layers noise.  Each layer takes all the signal from below that becomes their noise but detects new signals that they work to.  An upward flow - not downward instruction.  As the figure below shows, each layer has to apply different skills to find the signals needed to do their role.  Interestingly, the lowest layers have the most operations discretion as they are exposed to the highest operational risks.  Most diversity is welcome, but not from those who cannot do the same thing every time.  Innovation may not be your friend in the depths.  The strategy layer has peak decision discretion. Innovation is critical, and so is all diversity is critical. At the pinnacle is policy discretion, where is the least personal risk and whilst diversity is essential but so is adherence to a north star and single vision - so less diversity might help.  Diversity is about situational improvement in decision making for better outcomes.  Ouch.

However, this is not what we are witnessing right now - the above is theoretical rubbish.  What we are feeling is summed up in the diagram below.  There is are signals from the bottom, creating signals at the top, every layer adding more signals and noise.  VUCA has gone from situational layers to the entire organisation, where everyone is reacting to everything.  The ghosts of old processes and previous decisions are no longer limited to a layer, but everyone owns everything and has to react and understand. To repeat a previous line.  Today our hierarchies,  analogous to decision making,  have become paralysed, confused and ineffective as the volume of data and signals mixed with noise has risen to a level where our ability to know what is the right thing to do is broken. Currently, we have not transitioned to something that works. 

The observation took time, but the question for us all is, do we believe it, and if so, what can we do?  Do you believe it should be reflected on within the context of being in the club and how much you have already compromised on?  Our Millenials are not in the club and will not compromise, ask them.  

Our stability has vanished, and our tools have broken.

Even to a casual observer, we live in turbulent times, which is seen through an increase in VUCA. The difficultly facing boards who carry responsibility for their decisions cannot be overstated.  We have to deal with the ghosts of the past, the voice of the present and the spirits of the future.  The diagram below brings out some of the tensions and conflicts being faced as we struggle to determine what we are optimising for.  The two-axis are communication (said and unsaid) and status (know and unknown).  The unsaid is that which is not written or spoken and assumed, often to avoid more conflict.  

We have to deal with the ghosts of the past, the voice of the present and the spirits of the future.

Top right (said and known) This is the day to day operational aspects of the board and senior leadership team. There are two camps (supporter and action owner) at the board with 10 people involved in decision making. The focus is on management, KPI’s BSC and reporting.  There are known status with data, and everyone is able and capable of engagement in the topics. We are married to this quadrant as it is easy and pays the remuneration, and we are incentivised to focus on it.  This quadrant has traditionally depended on stability, but with VUCA disrupting that and the volume to signal and noise - we have to spend all our time here as that is all we have time for.

The bottom right is said and unknown.  In this situation, everyone has a view about the unknown resulting in 10 camps (personal opinions and experience) with the 10 people.  However, communication tends to be frank and honest. We go to this quadrant every now and again but quickly withdraw to safer grounds.  

The known and unsaid, top left.  The unsaid here are the assumptions that we all make about everyone else in the room and their viewpoints.  This time there is are 100 camps as we all assume about what everyone else thinks about everyone as there is no communication.  The principal reasons not to communicate is conflict, dominance and leadership style.  We are trapped by the debt we have (mortgage, credit, school fees, lifestyle, divorce) and need the salary; therefore, there is a degree of control.  We try to avoid this quadrant but every now and then, we end up there because of other stress, pressures and the need for a diversion or win a political game.  

The bottom left, the unknown and unsaid.  The ecosystem has a voice, the partners of the directors have a voice, everyone has a voice, and many, due to previous flights and allegiances, are in several camps at the same time.  The crux is that we are human and bring our differences, but it is very messy because it is unknown and unsaid. But this quadrant represents the volatility, uncertainty, complexity and ambiguity we are currently facing, and we don’t have the tools to deal with it. We have a preference based on skills, experience and incentives to focus on the top right. 

Our processes and methods enable only certain decisions

One critical aspect of being in a leadership capacity is to question and determine how our processes and methods guide and frame certain decisions, which means we have to unpack legacy and find the ghosts in the system.

Legacy within this framing is threefold. Decisions. Decisions. Decisions. These are:

Previous incentives and power games created decisions that created processes, methods and rules; they are now ghosts in the systems. These decisions were taken so long ago that no one knows why, how or when it was decided. It is the way we do it; it is our IP, our brand.

Decisions that created “information and technology debt” included embedded and baked-in systems, hidden and no-longer supported code, and automation based on tools and data biased when created.

Decisions that created noise in the hierarchy to lose or filter signals that someone did not want to hear. It was the creation of layers, reports, practices, structural regulation and unchallenged assumptions.

Unpacking legacy questions will take time. It is worth asking questions about legacy when you are new to a company and then verifying them over time as we become blind to the tools that mould us.

I am focused on this because I want to determine how I can make smarter/ better decisions with data. For that, I need a data decision framework.  Therefore I tend to ask what one thing do we as a leadership team want from our data? The response varies but include:

Evidence-based, actionable insights

What should we automate?

How do we know we are doing the right thing?

Where are there efficiencies to be gained?

What do customers really want?

How to manipulate customers to increase margin and revenues?

Where are risks that we cannot see?

What is being hidden that we cannot see?

If you look at this list in the context of what tools and decisions already frame the response, are these what we are looking for data to answer or are we looking to data to affirm/ justify what we have already decided. A different response that no one will say “to justify what we are already doing!”  This fits into the know/ unsaid in the previous matrix. The top left, the one we avoid opening up.  

Data has bias because of previous decisions.  Or we can write, “the ghost of previous decisions will guide our current decisions”.  Importantly, our data, which we trust, is only the representation of the past, which means our tools fail.

Therefore, as a leadership team, we have to find non-data tools to check what decisions from the past are biasing the current data, processes and tools. We cannot usefully answer the question we have set ourselves “What is the one thing that we, as a team and organisation what our data to drive, deliver or provide? Without understanding the situation.

The CTO knows that they have to build a new platform when the bug list, new feature development and maintenance costs are bigger and will take more time than developing a new platform — this is the technology debt question. The CIO or newly created #CDO role has to understand what is your information debt. The CTO will struggle as there is no clear path from policy to code. Similarly, the CIO/CDO struggles with no clear path from policy (what one thing) to better data for the decisions we require. The data leadership team inherit and are now accountable for previous ghosts and decisions, which constrain what is now possible as the biased tool has created what we have. The costs of collecting, labelling, holding, sorting and creating training data continually increase, creating a more significant gap and misalignment in values and expectations from data.

“We become what we behold. We shape our tools, and then our tools shape us” is often mistakenly attributed to Marshall McLuhan and called McLuhan Law. The quote was actually written by Father John Culkin, SJ, a Professor of Communication at Fordham University in New York and friend of McLuhan. Such is the problem with data.

As we have separate decisions, power and now data, perhaps we should reflect on these questions as a leadership team.

What do we want to become, and what tools and data will help us?

What tools do we use, and what will they enable us to become?

What is the minimum viable data set required to give the best value?

Do our tools and data trap us?

Is the work of data to “detect and measure” or to “enables change and transform?” 



The laws of 1% - how far before you reach a revolution?

What happens when we decrease life support by 1% Taking water down by 1% per day once you pass a threshold of low water intake and death is a certainty in the short term (blue line).  Taking food down by 1% per day, once you pass a threshold, death is a certainty, but it is slower to take you from life than reducing water (green line). Taking lifestyle or experience down by 1% per day gets
What happens when we decrease life support by 1%

Taking water down by 1% per day once you pass a threshold of low water intake and death is a certainty in the short term (blue line).  Taking food down by 1% per day, once you pass a threshold, death is a certainty, but it is slower to take you from life than reducing water (green line). Taking lifestyle or experience down by 1% per day gets more challenging and more complex, and you will likely die earlier, but death is not facing you (black line).  We all die, but lack of water means it happens now, lack of food means it happens soon, lack of lifestyle means life could have been longer (black line).  Leaving purple as the optimal.

When reducing water by 1%, you reach a point very quickly where revolution is worth it as there is nothing to lose.   When reducing food by 1%, you reach a point slowly where revolution is worth it as there is less to lose. Still, if you go too far beyond the point when you have agency and capacity from sufficient food, you lose the ability to win the revolution. And you have just enough; why give it up?  Reducing lifestyle by 1% revolution may happen when sufficient have nothing left to lose, which may never come. 

Taking away can lead to revolution. How to prevent revolution, improve by 1%

What happens when we increase life support by 1%

Increasing water by 1% per day, once you pass a threshold of high water intake, death is a certainty from drowning (blue line) but is very hard to enforce.  Increasing food by 1% per day, there is no threshold where death is a certainty because it is so slow. Increasing lifestyle or experience by 1% per day gets more straightforward, and you will die from something but death is not facing you (black line).  Whilst we all die, too much water, food or lifestyle puts death into the distance, and you have agency and choice. 

 

Making Stability look like 1% per year

When change happens at 1% per day, we notice, when does 1% looks like stability?  A week, month or year.  If I change water, food or lifestyle up or down by 1% per year, life will not change much. However, if down, you will reach a point in 20 years where you can smell revolution. If up, do we become comfortably numb and depend on hope that rather than change. 

Is growth just a mask for the prevention of a power change?  How much will you have to lose before a revolution?

If the climate changes up by 1% per year - how long will it take?  If the climate changes down by 1% - how long will it take?

Thursday, 03. June 2021

Hyperonomy Digital Identity Lab

Hydroponic Pods

Wednesday, 02. June 2021

Phil Windley's Technometria

SSI Interaction Patterns

Summary: Interaction patterns in SSI differ from what we see in a traditional IAM system that is performing authentication and authorization. This post discussed three different authentication and authorization patterns and then shows how they are all specializations of the general verifiable credential exchange pattern. Last year, I wrote about how digital relationships are operation

Summary: Interaction patterns in SSI differ from what we see in a traditional IAM system that is performing authentication and authorization. This post discussed three different authentication and authorization patterns and then shows how they are all specializations of the general verifiable credential exchange pattern.

Last year, I wrote about how digital relationships are operationalized in response to a post from Doc Searls about SSI wallets. The wallet (and the agent it is paired with) is a key player in SSI workflows. A recent exchange with the good folks at TechVision Research made me realize that I hadn't ever written about the patterns that an SSI wallet uses to realize operational digital relationships. Today I'm going to take a stab at three simple authentication and authorization patterns in SSI to show the interactions necessary to accomplish these foundational workflows. Finally, I'll show how all three are just specializations of the standard verifiable credential exchange pattern.

DID Authentication Pattern

The simplest authentication pattern uses decentralized identifiers (DIDs) as autonomic identifiers to establish a peer relationship. Because of their mutual authentication capabilities, DID relationships can be used for authentication.

Simple DID Authn Interaction Pattern (click to enlarge)

This pattern has two parties:

Alice has an SSI wallet on her mobile phone. Bravo Corp has an enterprise wallet tied to an IAM system that is protecting some resource.

The interaction pattern has the following steps:

Alice and Bravo establish a Peer DID relationship (blue arrow). This means that they each generate a Peer DID and send it to the other, along with the associated public key. These identifiers are self-certifying and each party can use the information associated with the DID to authenticate the other. Alice tries to access the protected resource (red arrow). The request is intermediated by Bravo's IAM system. As part of this request, Alice makes her DID known. There are a number of sub-scenarios for the different ways this may happen. For example, she could scan a QR code or enter an associated human-readable identifier. The IAM system, working in concert with Bravo's enterprise wallet, issues a DID Auth challenge to Alice's wallet through her phone. Alice is notified by her wallet of the challenge and approves the response from her wallet to Bravo. Bravo verifies Alice's response.

A few things to note about this interaction:

Because Alice and Bravo are using Peer DIDs, no ledger is involved in the authentication. In a Peer DID relationship, both parties keep the other informed of relevant key events (e.g. key rotation) and store that information in a cryptographic key event log. Any authorization would have to be done based on information the IAM system has from another source. For example, if the Peer DID relationship were established within a different authenticated context, Alice could have been assigned a group for RBAC or other attributes could have been associated with Alice's DID within Bravo's IAM system. The interaction pattern shown here is leaves out a number of details. Markus Sabadello identifies ten different variations of this pattern in his talk Introduction to DID Auth for SSI. Single-Party Credential Authorization Pattern

While the DID Authn pattern is simple, it is not as flexible as we need in some situations. For more complicated scenarios, we can use verifiable credentials. The first scenario we'll consider is where the same organization is issuing and verifying the credential.

Single-Party Credential-Based Authn Pattern (click to enlarge)

The parties in this scenario are the same: Alice and Bravo Corp.

The interaction pattern proceeds as follows:

Since Bravo Corp will be issuing a credential, they write a Public DID and credential definition to the ledger. They might also write a schema and revocation registry, if necessary. Alice and Bravo establish a Peer DID relationship (blue arrow). Note that the DID that Bravo uses for this relationship is not the public DID created in (1), instead Bravo creates a Peer DID especially for the relationship with Alice. Bravo issues a credential to Alice (green arrow). The nature, content, and context of this credential issuance depend on Bravo and Alice's specific needs. Bravo is the credential issuer and Alice is the credential holder. Alice tries to access a protected resource (red arrow). The request is intermediated by Bravo's IAM system. Like the DID Authn pattern, the IAM system is working in concert with an enterprise wallet. Bravo is using a policy-based access control (PBAC) system that relies on knowing attributes about Alice. The IAM system makes a credential request to Alice that asks for specific attributes based on the attributes needed by the policy for the resource Alice is accessing. Alice sees the request and authorizes her wallet to issue a proof of attributes based on the credential she holds. The response contains only the attributes that Bravo needs, not the entire credential to minimize the information that is shared. The PBAC system uses the attributes in the proof presentation to authorize Alice's access.

A few things to note:

Bravo does not need to access the ledger to verify the credential since they already know the information necessary to perform the validation since it's their credential. Even so, Bravo writes the public DID and credential definition to the ledger so that Alice can present the credential to others who can verify it, supporting use cases beyond Bravo's. Using a credential held by Alice to validate her authority to access the protected resource is more flexible for Bravo, and potentially more reliable, than a centralized attribute store. Rather than building a central attribute store and linking every system in the enterprise to it, each system can stand alone from the central store and make decisions based on the policies in place for that system. Astute readers will read the last bullet and think "but don't they all have to be linked to the same digital wallet to take advantage of the Peer DID relationship?" The answer is "no." Each service can have its own Peer DID relationship with Alice, verify the attributes from the credential, and know it's Alice. The only thing they need to know is the public DID their organization uses and the credential definition for the credential. Multi-Party Credential Authorization Pattern

We can extend the single-party pattern, to include multiple parties. In this pattern, one entity, Bravo Corp, is issuing credentials, but another entity, Certiphi Corp, is verifying the credential and using its attributes to authorize Alice's access to a resource.

Multi-Party Credential-Based Authn Pattern (click to enlarge)

The interaction proceeds as follows:

Since Bravo Corp is issuing a credential, they write a Public DID and credential definition to the ledger. Again, they might also write a schema and revocation registry, if needed. Alice and Bravo establish a Peer DID relationship (blue arrow). Bravo issues a credential to Alice (green arrow). Alice and Certiphi establish a Peer DID relationship. Alice tries to access the protected resource (red arrow) at Certiphi. The request is intermediated by Certiphi's IAM system. Certiphi is using a policy-based access control system so, the IAM system makes a credential request to Alice that asks for the specific attributes needed by the policy for access to the resource. Alice sees the request and authorizes her wallet to issue a proof of attributes based on the credentials she holds. The wallet automatically chooses the credential from Bravo since it has the attributes needed to satisfy Certiphi's request. Certiphi cryptographically validates the fidelity of the proof to ensure it's from Bravo, is about Alice, hasn't been tampered with, and hasn't been revoked. They might also need to validate the provenance of the attributes in the proof. Certiphi is the credential verifier in this pattern. The PBAC system uses the attributes in the proof presentation to authorize Alice's access.

A few things to note:

The DID relationship Alice and Certiphi create in (4) could be ephemeral, it needn't be permanent unless the parties need it to be. There is no direct connection or link between Bravo Corp and Certiphi Corp. They needn't have any pre-existing business or technical relationship. Certiphi needn't connect to a Bravo Corp API. The primary difference between the single-party and multi-party patterns is step (8), checking the fidelity and provenance of the credential. The fidelity can be done automatically using cryptography. Determining provenance is not a trivial thing since it involves Certiphi determining they can trust attributes attested by Bravo. This is a governance issue, not a technical one. The governance issue could be simple or complex. Perhaps Bravo is known to Certiphi (e.g., a local business next to a large university). Certiphi might ask Bravo to prove things about itself using credentials issued to Bravo by someone Certiphi already trusts (e.g., the government). Bravo and Certiphi might already be part of some established governance framework (e.g., a university accreditation organization). Generalized Trustworthy Data Transfer Pattern

Authentication and authorization are table stakes for any identity interaction. The general data transfer pattern moves beyond simple authentication and authorization patterns to using identity data in workflows.

Credential-Based Data Transfer Pattern (click to enlarge)

In this pattern, all of the interactions are identical to the pattern for multi-party authorization in the last section with a few exceptions:

Alice is accessing a web service that needs data to proceed in (5) rather than a protected resource. The web service uses the data from the proof presentment as part of its workflow (e.g. fill out a form).

We can view all of the previous patterns as specializations of this pattern:

The Peer DID relationship provides a mutually authenticated communications channel in every case that can always be used to know that you're talking to the entity with whom the relationship was originally established–the core requirement for any authentication. Transferring attributes using verifiable credentials for PBAC is just a special case of transferring attribute data in a trustworthy manner. The difference is the end-use of the attributes: the PBAC system or some other service. There's no need for the data transferred in the general pattern to come from a single credential. In fact, the service can ask for attributes without knowing what credentials Alice holds. Alice's wallet will match the requested attributes to the credentials Alice holds. Alice can choose which credentials to use for specific attributes (e.g. date of birth) if she wants. While the figure shows Alice accessing a web service, this can be further generalized beyond the web. Any data transfer for an online workflow can happen using verifiable credentials. While this pattern involves Alice and two organizations, there's no reason why people can't be credential issuers and verifiers. Indeed, any party in these diagrams could play any of the roles.

Viewing traditional IAM functions like authentication and authorization as special purpose data transfers broadens SSI significantly beyond what we have traditionally seen as "digital identity." The uses for verifiable credentials are vast and include many things we may not think of as "credentials". While this expanded view of digital identity may make some uncomfortable, I think it is perfectly aligned with my belief that we build identity systems to manage relationships, not identities. Every relationship is unique. Flexible, trustworthy digital credentials serve that uniqueness and introduce the means of moving digital identity beyond just authentication and authorization.

Photo Credit: Phone Icon from Fast Icon Design (Linkware)

Tags: identity ssi decentralized+identifiers verifiable+credentials


Jon Udell

Dear background mind: please think about Postgres and Clojure

I used a Lisp variant in my first programming job, so I have some appreciation for the “code as data” power of that language. Nowadays I’m building an analytics system that combines Postgres and two of its procedural languages, pl/pgsql and pl/python, with a sense-making tool called Metabase that’s written in a Lisp variant called … Continue reading Dear background mind: please think about Postgres

I used a Lisp variant in my first programming job, so I have some appreciation for the “code as data” power of that language. Nowadays I’m building an analytics system that combines Postgres and two of its procedural languages, pl/pgsql and pl/python, with a sense-making tool called Metabase that’s written in a Lisp variant called Clojure.

In Postgres I’m able to wrap SQL queries in functions; they compose with other functions that do things like cache results in materialized views and aggregate subgroups. It all feels very dynamic and functional, which are two of Clojure’s main calling cards, so this makes me wonder about Clojure as another Postgres procedural language.

For the pl/python functions in my system, “code as data” looks like building SQL statements that swap variables into SQL templates and combine them. This string-building approach is an anti-pattern for Clojure folk. They don’t want to work with fragments of SQL text, they want to work with Clojure maps. Here’s an example from Honey SQL, the library that Metabase uses to build SQL texts from structured data.

(def sqlmap {:select [:a :b :c]
:from [:foo]
:where [:= :f.a "baz"]})

This seems like an important power to be able to wield in the hybrid programming environment that Postgres provides. I can imagine making very good use of it.

But I know very little yet about Clojure. Is this really an idea worth exploring? How else would it differ from what I’m now able to do in Python? To learn more about Clojure I’ve been watching talks by its creator, Rich Hickey. Most are tech-heavy but one of them isn’t like the others. In Hammock-driven development he lays out a process for creative problem solving that coordinates the waking mind (at the computer doing critical thinking and analysis) with the background mind (on the hammock doing abstraction, analogy, and synthesis). The coordination is explicit: You use the waking mind to feed work to the background mind which is “the solver of most non-trivial problems”; you weight your inputs to the background mind in order to influence their priority in its thinking.

I guess I’ve done that kind of thing implicitly from time to time, but never in such an intentional way. So, is Clojure-in-Postgres worth exploring? Perhaps by writing this I’ll prime my background mind and will receive more clarity in the morning.

Tuesday, 01. June 2021

MyDigitalFootprint

The Paradox that war and monopoly are the same thing.

Peak Paradox is a framework to help you reclaim clarity and control as we believe that if you cannot see the paradox, you have been framed and are in a model where someone else is in control.  When you can see the paradox in the data, information, recommendation or decision, you will have improved your clarity.  This is a guide on how to apply the framework to unlock business models, co

Peak Paradox is a framework to help you reclaim clarity and control as we believe that if you cannot see the paradox, you have been framed and are in a model where someone else is in control.  When you can see the paradox in the data, information, recommendation or decision, you will have improved your clarity.  This is a guide on how to apply the framework to unlock business models, concepts and complexity.

Most aspects of everyday business and life can be questioned within the boundaries of the Peak Paradox Framework.  I have been exploring that we tend to go out of bounds for a while but find it is too hard to stay there for long. Movement outside of the boundary tends to be a transitional state.  The thick red line illustrates the usual boundaries in the figure below.  This post explored how to make better decisions using the framework and laid down the idea that some decisions can occur off-piste, outside of the normal boundaries. It presented the reason why we need to minimise shared decision making, find followers and create shared passion if we want to make a difference.  Indeed it is no longer about enemies in the camp but about identifying those who share the belief. We must have diversity and not about diversification away from the shared belief.  

I explored the concept that we are transitioning from Physical beings to Information beings.  The article proposed that our existing models for oversight and governance are breaking and that we need to think about addressing this emerging gap.  In an earlier post, I explained that many models take us to the boundaries in the Peak Paradox framework, such as cults, politics, beliefs and religion.  This post presents the thinking that the economic model of monopoly and political activity of war are outside the framework and a transitional state. 

As physical beings 

In the figure below, war and monopoly are outside the boundary between Peak Individual Purpose and Peak Work Purpose.  War is nasty, inhumane, and complex, but it serves as a tool to acquire more land with a historical lens. Control of land gave you access to resources, people and materials.  The more land and more resources you had access to, in straightforward terms, enabled a few to realise their Peak Individual Purpose (become more sovereign) and for others to realise Peak Work Purpose through growth. 

A more modern version of war is the economics of monopoly, duopoly, or cartel, where a few seek to control resources for their personal gain, enabling them to achieve Peak Individual Purpose through their companies having control.  

Both war and monopoly have become increasingly regulated over time. We have created, as humans, complex rules of engagement, governance and oversight.  In part to ensure control and power does not fall to one person in a democracy, to protect the vulnerable and to respect life.  If they work or not is another story. 

As information beings

It is not that physical has or will vanish; we will co-exist as both physical and informational. Our physical being is still very much here, but we live in an emergent digital/ data augmented world.  War has become cyberwarfare and monopoly of land, and people have become the monopoly of data that indirectly controls people and land, often through an instrument such as debt. 

The question being asked is, “have we changed more than just the nature of war and monopoly as we expand from physical beings to information beings?”  Our rules, methods, insights and policy work for us as physical beings, but do the same systems and structures work in an information age.  Is the paradox that we have to deal with the same outcomes from war and monopolies (control and power), but there is a shift.  What is this shift?   Yes, some cybercrime and cyber warfare is a replication of the historical war framing, insomuch that an individual, cause or nation wants to use data/ digital as a method of control.  But this is just the natural advancement of physical beings adapting to new technology. Gunpowder is another example.  However, information being cyberwar is a fight or rage against the machine trying to bring about a better life for everyone and expose injustice - this is different. Past freedom fighters against oppression look similar.  A monopoly of data is different to a monopoly of physical resources because data is data.  Data is, by its nature, non-rivalrous and non-excludable. This simple fact is that you cannot declare ownership of data (though many people try); you cannot control it; you lose nothing when you copy it. It is why data is data.   Data can bring about better health care; data can bring about better education; data can bring transparency, and data can bring equality.  

An AI can make better decisions, if we knew what this is, without the bias of human politics. However, it is far too easy to keep our physical being lens on everything as we like it, we are used to it, our systems are aligned to it, and we can ignore or rubbish the emergence and augmentation of data in our decision methodology.  A critical point beginning that what we had as oversight and controls for war and monopoly worked for physical beings. The same systems break as informational beings because there is a shift in purpose and even outcomes. What is hard to comprehend is that this is not a transition; it is the emergence of information beings and that both will co-exist. We need to keep adapting our physical rules but need new thinking for informational beings.    

What does the Peak Paradox framework highlight about this situation? There is an assumption that we are aligned (what type of being are we or both) and how we understand if there is alignment in the team.  Because this is hard and subtle, we prefer to assume and make jumps because curiosity and questioning expose ghosts in our thinking and the system;  like conflict, tension and power. We don't like facing up to these as they are hard and reflective. The Peak Paradox framework will surface the ghosts and determine if there is an alignment or a delta in the team, project, mission or purpose.  If we are only physical beings in our views, we will ignore digital and data and the coming information being.   If we are information beings, we see that the controls don’t work, and we are free to exploit an emergent space.  The current data monopoly holders don’t look like monopolies through the lens of physical beings.

The current data monopoly holders don’t look like monopolies through the lens of physical beings.    

War and monopoly exist in both the physical and data/ digital/ information world. Still, we cannot apply the same system of the physical world controls developed over generations to the digital/ data/ information world - it is not they break it is they don’t work.   The system of control fails because the purpose for which war and monopoly exist is the same and has created a new space.  We use the exact same words, but the meaning has changed.  Why is monopoly different in a data/ digital information world? Because it looks competitive, data is movable, and users have a choice.  Why is war in a data/ digital information world different - because the outcome is not about control.  However, the physical still exists, which can confuse and create paradoxes.

 


Monday, 31. May 2021

Doc Searls Weblog

Comparing cameras

On the top left is a photo taken with my trusty old (also much used and abused) Canon 5D Mark III. On the top right is one taken by a borrowed new Sony a7Riii. Below both are cropped close-ups of detail. The scene is in a room illuminated by incandescent track lighting. It is not […]

On the top left is a photo taken with my trusty old (also much used and abused) Canon 5D Mark III. On the top right is one taken by a borrowed new Sony a7Riii. Below both are cropped close-ups of detail. The scene is in a room illuminated by incandescent track lighting. It is not an art shot, though it does contain photo art by our good friend Marian Crostic, whose Sony a7R she is kindly remanding to my custody tomorrow. (Her main camera is now an a7Riii like the borrowed one I used here.)

Both photos were shot with Canon and Sony’s best 24-105 f4 zoom lenses, at the 105mm end. Both were also set to automatic, meaning the camera chooses all the settings. In both cases the camera chose ISO 3200 at f4. The only difference was shutter speed: 1/125 sec on the Canon and 1/160 sec on the Sony. While 3200 is not the prettiest ISO, I wanted to compare both cameras indoors under less than ideal lighting, because that’s typical of situations where I shoot a lot of people.

One difference between these cameras is the pixel density of the sensor: the Canon’s shot is 5760 x 3840 pixels, while the Sony’s is 7952 x 5304. While that difference accounts for some of the higher detail in the Sony’s shot, it’s clear to me that the Sony lens is simply sharper, as Ken Rockwell kinda promised in this glowing review. (Also, to be fair, the Canon lens has had a lot of use.)

All the images above are screen shots of RAW versions of the photos (.CR2 for the Canon and .ARW for the Sony). Though I don’t have the time or patience to show differences in the .JPG versions of these photos, it’s clear to me that the Canon’s JPGs look less artifacted by compression. The obvious artifacts in the Sony shots have me thinking I may only shoot RAW with the a7R, though I’ll need to test it out first.

The main difference overall, at least in this setting, is in the warmth of the color. There the Canon has a huge advantage. I could say it’s also because the Sony is slightly less exposed (by the higher shutter speed); but I noticed the same difference in test shots I took outdoors as well, under both overcast and sunlit skies, and at ISO 100. The Canon seems warmer, though the Sony has far more detail one can pull out of shadows.

I should add that neither camera got the color of the wall (a creamy white) right in these photos, with the Canon leaning hot and the Sony leaning cool.

Anyway, I just thought I’d share that much before I pick up the a7R, and start using it to shoot stuff in New York, where I’m headed Wednesday night after more than a year away.

 

 


Damien Bod

Verify vaccination data using Zero Knowledge Proofs with ASP.NET Core and MATTR

This article shows how Zero Knowledge Proofs ZKP verifiable credentials can be used to verify a persons vaccination data implemented in ASP.NET Core and MATTR. The ZKP BBS+ verifiable credentials are issued and stored on a digital wallet using a Self-Issued Identity Provider (SIOP) and Open ID Connect. The data can then be used to […]

This article shows how Zero Knowledge Proofs ZKP verifiable credentials can be used to verify a persons vaccination data implemented in ASP.NET Core and MATTR. The ZKP BBS+ verifiable credentials are issued and stored on a digital wallet using a Self-Issued Identity Provider (SIOP) and Open ID Connect. The data can then be used to verify if the holder has the required credentials, but only the required data is used and returned to the verification application. The holder of the data who owns the wallet can consent or not consent to allow the verification application to see and use the vaccination data. Auth0 is used to implement the identity provider.

Code https://github.com/swiss-ssi-group/MattrZeroKnowledgeProofsAspNetCore

Blogs in the series

Getting started with Self Sovereign Identity SSI Create an OIDC credential Issuer with MATTR and ASP.NET Core Present and Verify Verifiable Credentials in ASP.NET Core using Decentralized Identities and MATTR Verify vaccination data using Zero Knowledge Proofs with ASP.NET Core and MATTR

What are Zero Knowledge Proof enabled credentials

Zero Knowledge Proof enabled credentials allows you to selectively disclose claims from a verifiable credential without disclosing all of the information of the verifiable credential. It also makes it possible to verify data without having to share the sensitive data required to verify something. In this post, we will just selectively request part of the data from a verifiable credential. This would make it possible to implement business flows without having to share or copy the sensitive data.

Setup ZKP Vaccination Data Issue and Verify

The demo application implements a covid vaccination data process. A number of components, applications are required to implement these flows. The idea is that when a person is vaccinated, the authority responsible for this could add the vaccination data to the persons identity. In this demo, that would be added to the Auth0 service which can be accessed in the id_token claims. The vaccination data organization can use the credentials issuer application to create a DID credential issuer using ZKP verifiable credentials. The end user can use a digital wallet to add his or his credentials using the SIOP flow which gets the claims from the IDP and adds the data to the digital wallet. The verification application would have to create a presentation template defining the claims which are required to use to verify. Once created, a new verification request can be created and used to verify the vaccination data from the user. The user would scan the presented QR Code from the verifier application and display a verify in the digital wallet. Once consented, the data is returned to the verifier application using an API. The data can be processed and the UI is updated with a verified result or not. The blockchain ledger is abstracted away and used indirectly through MATTR which has APIs for the Self sovereign identity specifications.

Issuing Zero Knowledge Proof enabled credentials

The VaccineCredentialsIssuer ASP.NET Core application is used to create the credential issuer and present this as a QR Code for the user to add vaccination Zero Knowledge Proofs verifiable credentials. The flow implemented is very similar to the flow used in the previous blog Create an OIDC credential Issuer with MATTR and ASP.NET Core . A DID is created to use a BLS key type which supports BBS+ signatures for issuing ZKP-enabled credentials.

public class MattrOptions { /// <summary> /// The supported key types for the DIDs are ed25519 and bls12381g2. /// If the keyType is omitted, the default key type that will be used is ed25519. /// /// If the keyType in options is set to bls12381g2 a DID will be created with /// a BLS key type which supports BBS+ signatures for issuing ZKP-enabled credentials. /// </summary> public string keyType { get; set; } = "bls12381g2"; }

The DID is used to create a credential issuer for the ZKP credentials. The CreateMattrCredentialIssuer method takes the DID created with the bls12381g2 key and creates the OIDC credential issuer.

private async Task<V1_CreateOidcIssuerResponse> CreateMattrCredentialIssuer(HttpClient client, V1_CreateDidResponse did) { // create vc, post to credentials api // https://learn.mattr.global/tutorials/issue/oidc-bridge/setup-issuer var createCredentialsUrl = $"https://{_mattrConfiguration.TenantSubdomain}/ext/oidc/v1/issuers"; var payload = new MattrOpenApiClient.V1_CreateOidcIssuerRequest { Credential = new Credential { IssuerDid = did.Did, Name = "VaccinationCertificate7", Context = new List<Uri> { new Uri( "https://schema.org"), new Uri( "https://www.w3.org/2018/credentials/v1") }, Type = new List<string> { "VerifiableCredential" } }, ClaimMappings = new List<ClaimMappings> { new ClaimMappings{ JsonLdTerm="family_name", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/family_name"}, new ClaimMappings{ JsonLdTerm="given_name", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/given_name"}, new ClaimMappings{ JsonLdTerm="date_of_birth", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/date_of_birth"}, new ClaimMappings{ JsonLdTerm="medicinal_product_code", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/medicinal_product_code"}, new ClaimMappings{ JsonLdTerm="number_of_doses", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/number_of_doses"}, new ClaimMappings{ JsonLdTerm="total_number_of_doses", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/total_number_of_doses"}, new ClaimMappings{ JsonLdTerm="vaccination_date", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/vaccination_date"}, new ClaimMappings{ JsonLdTerm="country_of_vaccination", OidcClaim=$"https://{_mattrConfiguration.TenantSubdomain}/country_of_vaccination"} }, FederatedProvider = new FederatedProvider { ClientId = _configuration["Auth0Wallet:ClientId"], ClientSecret = _configuration["Auth0Wallet:ClientSecret"], Url = new Uri($"https://{_configuration["Auth0Wallet:Domain"]}"), Scope = new List<string> { "openid", "profile", "email" } } }; var payloadJson = JsonConvert.SerializeObject(payload); var uri = new Uri(createCredentialsUrl); using (var content = new StringContentWithoutCharset(payloadJson, "application/json")) { var createOidcIssuerResponse = await client.PostAsync(uri, content); if (createOidcIssuerResponse.StatusCode == System.Net.HttpStatusCode.Created) { var v1CreateOidcIssuerResponse = JsonConvert.DeserializeObject<V1_CreateOidcIssuerResponse>( await createOidcIssuerResponse.Content.ReadAsStringAsync()); return v1CreateOidcIssuerResponse; } var error = await createOidcIssuerResponse.Content.ReadAsStringAsync(); } throw new Exception("whoops something went wrong"); }

The data from the MATTR response is used to create the callback for the credentials issuer. This is persisted to a database as this needs to be created only once and can be re-used.

public async Task<string> CreateCredentialsAndCallback(string name) { // create a new one var vaccinationDataCredentials = await CreateMattrDidAndCredentialIssuer(); vaccinationDataCredentials.Name = name; await _vaccineCredentialsIssuerCredentialsService.CreateVaccinationData(vaccinationDataCredentials); var callback = $"https://{_mattrConfiguration.TenantSubdomain}/ext/oidc/v1/issuers/{vaccinationDataCredentials.OidcIssuerId}/federated/callback"; return callback; }

The data is displayed as a QR Code in a ASP.NET Core Razor page application. This can be scanned and the credentials will be added to your digital wallet, if the Open ID Connect server has the claims for the identity required by this issuer.

Auth0 is used to add the identity data for the claims. An Auth0 pipeline rule was created and used to add these claims to the id_tokens.

function (user, context, callback) { const namespace = 'https://damianbod-sandbox.vii.mattr.global/'; context.idToken[namespace + 'date_of_birth'] = user.user_metadata.date_of_birth; context.idToken[namespace + 'family_name'] = user.user_metadata.family_name; context.idToken[namespace + 'given_name'] = user.user_metadata.given_name; context.idToken[namespace + 'medicinal_product_code'] = user.user_metadata.medicinal_product_code; context.idToken[namespace + 'number_of_doses'] = user.user_metadata.number_of_doses; context.idToken[namespace + 'total_number_of_doses'] = user.user_metadata.total_number_of_doses; context.idToken[namespace + 'vaccination_date'] = user.user_metadata.vaccination_date; context.idToken[namespace + 'country_of_vaccination'] = user.user_metadata.country_of_vaccination; callback(null, user, context); }

The data needs to be added to each user in Auth0. If using this in a real application, a UI could be created and used to add the specific data for each user. The credential issuer is tightly coupled through the data with the IDP. So each credential issuer which creates verifiable credentials would require it’s own identity provider and full access to update the profiles. The IDP contains the business data required to issuer the credentials. Auth0 might not be a good choice for this, maybe something like IdentityServer or Openiddict would be a better choice because you could implement custom UIs with ASP.NET Core Identity and the complete UIs for the credential issuing flows.

When the credential issuer is scanned by the digital wallet, the user logs into the OIDC server and gets the data for the ZKP verifiable credentials. In a MATTR wallet, this is displayed with the Privacy enhancing credential information.

Verifying the credentials

Before the ZKP credentials can be verified, a presentation template is created to define the required credentials to verify. The DID ID from the credential issuer is used to find the DID in the ledger. The CreateMattrPresentationTemplate method creates the template using the QueryByFrame so that the exact claims can be defined. The context must use the https://w3c-ccg.github.io/ldp-bbs2020/context/v1 namespace to use the ZKP BBS+ credentials in MATTR. The type must be VerifiableCredential.

private async Task<V1_PresentationTemplateResponse> CreateMattrPresentationTemplate( HttpClient client, string didId) { // create presentation, post to presentations templates api // https://learn.mattr.global/tutorials/verify/presentation-request-template // https://learn.mattr.global/tutorials/verify/presentation-request-template#create-a-privacy-preserving-presentation-request-template-for-zkp-enabled-credentials var createPresentationsTemplatesUrl = $"https://{_mattrConfiguration.TenantSubdomain}/v1/presentations/templates"; var additionalPropertiesCredentialSubject = new Dictionary<string, object>(); additionalPropertiesCredentialSubject.Add("credentialSubject", new VaccanationDataCredentialSubject { Explicit = true }); var additionalPropertiesCredentialQuery = new Dictionary<string, object>(); additionalPropertiesCredentialQuery.Add("required", true); var additionalPropertiesQuery = new Dictionary<string, object>(); additionalPropertiesQuery.Add("type", "QueryByFrame"); additionalPropertiesQuery.Add("credentialQuery", new List<CredentialQuery2> { new CredentialQuery2 { Reason = "Please provide your vaccination data", TrustedIssuer = new List<TrustedIssuer>{ new TrustedIssuer { Required = true, Issuer = didId // DID use to create the oidc } }, Frame = new Frame { Context = new List<object>{ "https://www.w3.org/2018/credentials/v1", "https://w3c-ccg.github.io/ldp-bbs2020/context/v1", "https://schema.org", }, Type = "VerifiableCredential", AdditionalProperties = additionalPropertiesCredentialSubject }, AdditionalProperties = additionalPropertiesCredentialQuery } }); var payload = new MattrOpenApiClient.V1_CreatePresentationTemplate { Domain = _mattrConfiguration.TenantSubdomain, Name = "zkp-certificate-presentation-11", Query = new List<Query> { new Query { AdditionalProperties = additionalPropertiesQuery } } }; var payloadJson = JsonConvert.SerializeObject(payload); var uri = new Uri(createPresentationsTemplatesUrl); using (var content = new StringContentWithoutCharset(payloadJson, "application/json")) { var presentationTemplateResponse = await client.PostAsync(uri, content); if (presentationTemplateResponse.StatusCode == System.Net.HttpStatusCode.Created) { var v1PresentationTemplateResponse = JsonConvert .DeserializeObject<MattrOpenApiClient.V1_PresentationTemplateResponse>( await presentationTemplateResponse.Content.ReadAsStringAsync()); return v1PresentationTemplateResponse; } var error = await presentationTemplateResponse.Content.ReadAsStringAsync(); } throw new Exception("whoops something went wrong"); }

The VaccanationDataCredentialSubject class defines the specific claims to use for the verification.

public class VaccanationDataCredentialSubject { [Newtonsoft.Json.JsonProperty("@explicit", Required = Newtonsoft.Json.Required.Always)] public bool Explicit { get; set; } [Newtonsoft.Json.JsonProperty("family_name", Required = Newtonsoft.Json.Required.Always)] [System.ComponentModel.DataAnnotations.Required] public object FamilyName { get; set; } = new object(); [Newtonsoft.Json.JsonProperty("given_name", Required = Newtonsoft.Json.Required.Always)] [System.ComponentModel.DataAnnotations.Required] public object GivenName { get; set; } = new object(); [Newtonsoft.Json.JsonProperty("date_of_birth", Required = Newtonsoft.Json.Required.Always)] [System.ComponentModel.DataAnnotations.Required] public object DateOfBirth { get; set; } = new object(); [Newtonsoft.Json.JsonProperty("medicinal_product_code", Required = Newtonsoft.Json.Required.Always)] [System.ComponentModel.DataAnnotations.Required] public object MedicinalProductCode { get; set; } = new object(); }

Verifying is very similar to the blog Present and Verify Verifiable Credentials in ASP.NET Core using Decentralized Identities and MATTR. A new DID of type ed25519 is used to invoke a verify request and also sign the request. The verifying flow in the application presents a QR Code using the redirectURL technic because the signed request is too long to present as a QR Code. This request returns a 302 with the full jws.

The application needs to be started using a public domain because the digital wallet will request back to the API with the data. I use ngrok to test locally. The verifier application can be started and the verify process is started by clicking the verify button which displays the QR Code to verify.

Start the application and start ngrok

ngrok http http://localhost:5000

The QR Code can be scanned to verify.

In the digital wallet, the verification request for the vaccination data can be viewed and if ok sent. The digital wallet displays the data which is disclosed and the data which is not. When the user clicks send, the data is validated and the API from the verifier application is called.

When the Verify application receives the callback from the digital wallet, the data is validated and the challenge ID is used to notify the user of a successful verification. The data is saved to a database and ASP.NET Core SignalR is used to update the UI. When the message from SignalR is sent, the user is redirected to the success page using the challenge ID and the data is displayed with the success image.

Notes

Now we have a full create, holder, verify process implemented for Zero Knowledge Proof verifiable credentials using covid vaccination data. OIDC is used to authenticate and create the claims used for the credentials. The OIDC connect server or identity provider is tightly coupled to the credential issuer because business uses the data from the id_token. When using SIOP, I would use ASP.NET Core Identity and either OpenIddict of Identityserver4 to implement this as part of the credential issuer. You need full control of the claims so using Auth0, Azure AD or Azure B2C would probably be a bad choice here. You could federate then to one of these from the credential issuer to use the profiles as required. Each vaccinated user would also require a user account. ZKP verifiable credentials makes it possible to support user privacy better and mix claims easier from different credentials. Another problem with this solution is the vendor lockdown. This is a problem with any self sovereign solution at the moment. Even though, the specifications are all standard, unless you want to implement this completely yourself, you would choose a vendor specific implementation which locks you down to a specific wallet or with specific features only. Interop does not seem to work at the moment. This is a problem with all security solutions at present not just SSI, all software producers, security services use security reasoning as an excuse to try to force you into lockdown into their specific product. You can see this with most of the existing OIDC solutions and services. Typical quotes for this are “You can use any OIDC client, but we recommend our OIDC client…” SSI will open possibilities for new security solutions and it will be very interesting to see how application security develops in the next five years.

Links

https://mattr.global/

https://learn.mattr.global/tutorials/verify/using-callback/callback-e-to-e

https://mattr.global/get-started/

https://learn.mattr.global/

https://keybase.io/

Generating a ZKP-enabled BBS+ credential using the MATTR Platform

https://learn.mattr.global/tutorials/dids/did-key

https://gunnarpeipman.com/httpclient-remove-charset/

https://auth0.com/

Where to begin with OIDC and SIOP

https://anonyome.com/2020/06/decentralized-identity-key-concepts-explained/

Verifiable-Credentials-Flavors-Explained

Sunday, 30. May 2021

Doc Searls Weblog

Apple vs (or plus) Adtech, Part II

My post yesterday saw action on Techmeme (as I write this, it’s at #2) and on Twitter (from Don Marti, Augustine Fou, et. al.), and in thoughtful blog posts by John Gruber in Daring Fireball and Nick Heer in Pixel Envy. All pushed back on at least some of what I said. Here are some […]

My post yesterday saw action on Techmeme (as I write this, it’s at #2) and on Twitter (from Don Marti, Augustine Fou, et. al.), and in thoughtful blog posts by John Gruber in Daring Fireball and Nick Heer in Pixel Envy. All pushed back on at least some of what I said. Here are some excerpts, with my responses. First, John:

Doc Searls:

Here’s what’s misleading about this message: Felix would have had none of those trackers following him if he had gone into Settings → Privacy → Tracking, and pushed the switch to off […].

Key fact: it is defaulted to on. Meaning Apple is not fully serious about privacy. If Apple was fully serious, your iPhone would be set to not allow tracking in the first place. All those trackers would come pre-vaporized.

For all the criticism Apple has faced from the ad tech industry over this feature, it’s fun to see criticism that Apple isn’t going far enough. But I don’t think Searls’s critique here is fair. Permission to allow tracking is not on by default — what is on by default is permission for the app to ask. Searls makes that clear, I know, but it feels like he’s arguing as though apps can track you by default, and they can’t.

But I don’t think Searls’s critique here is fair. Permission to allow tracking is not on by default — what is on by default is permission for the app to ask. Searls makes that clear, I know, but it feels like he’s arguing as though apps can track you by default, and they can’t.

I’m not arguing that. But let’s dig down a bit on all this.

What Apple has here is a system for asking in both directions (apps asking to track, and users asking apps not to track). I think this is weird and unclear, while simply disallowing tracking globally would be clear. So would a setting that simply turns off all apps’ ability to track. But that’s not what we have.

Or maybe we do.

To review… in Settings—>Privacy—>Tracking, is a single OFF/ON switch for “Allow Ads to Request to Track.” It is by default set to ON. (I called AppleCare to be sure about this. The guy I spoke to said yes, it is.) Below that setting is a bit of explanatory text with a “Learn more” link that goes to this long column of text one swipes down four times (at least on my phone) to read:

Okay, now look in the fifth paragraph (three up from where you’re reading now). There it says that by turning the setting to OFF, “all apps…will be blocked from accessing the device’s Advertising Identifier.” Maybe I’m reading this wrong, but it seems plain to me that this will at least pre-vaporize trackers vectored on the device identifier (technically called IDFA: ID For Advertisers).

After explaining why he thinks the default setting to ON is the better choice, and why he likes it that way (e.g. he can see what apps want to track, surprisingly few do, and he knows which they are), John says this about the IDFA:

IDFA was well-intentioned, but I think in hindsight Apple realizes it was naive to think the surveillance ad industry could be trusted with anything.

And why “ask” an app not to track? Why not “tell”? Or, better yet, “Prevent Tracking By This App”? Does asking an app not to track mean it won’t?

This is Apple being honest. Apple can block apps from accessing the IDFA identifier, but there’s nothing Apple can do to guarantee that apps won’t come up with their own device fingerprinting schemes to track users behind their backs. Using “Don’t Allow Tracking” or some such label instead of “Ask App Not to Track” would create the false impression that Apple can block any and all forms of tracking. It’s like a restaurant with a no smoking policy. That doesn’t mean you won’t go into the restroom and find a patron sneaking a smoke. I think if Apple catches applications circumventing “Ask App Not to Track” with custom schemes, they’ll take punitive action, just like a restaurant might ask a patron to leave if they catch them smoking in the restroom — but they can’t guarantee it won’t happen. (Joanna Stern asked Craig Federighi about this in their interview a few weeks ago, and Federighi answered honestly.)

If Apple could give you a button that guaranteed an app couldn’t track you, they would, and they’d label it appropriately. But they can’t so they don’t, and they won’t exaggerate what they can do.

On Twitter Don Marti writes,

Unfortunately it probably has to be “ask app not to track” because some apps will figure out ways around the policy (like all mobile app store policies). Probably better not to give people a false sense of security if they are suspicious of an app

—and then points to P&G Worked With China Trade Group on Tech to Sidestep Apple Privacy Rules, subtitled “One of world’s largest ad buyers spent years building marketing machine reliant on digital user data, putting it at odds with iPhone maker’s privacy moves” in The Wall Street Journal. In it is this:

P&G marketing chief Marc Pritchard has advocated for a universal way to track users across platforms, including those run by Facebook and Alphabet Inc.’s Google, that protects privacy while also giving marketers information to better hone their messages.

Frustrated with what it saw as tech companies’ lack of transparency, P&G began building its own consumer database several years ago, seeking to generate detailed intelligence on consumer behavior without relying on data gathered by Facebook, Google and other platforms. The information is a combination of anonymous consumer IDs culled from devices and personal information that customers share willingly. The company said in 2019 that it had amassed 1.5 billion consumer identifications world-wide.

China, where Facebook and Google have a limited presence, is P&G’s most sophisticated market for using that database. The company funnels 80% of its digital-ad buying there through “programmatic ads” that let it target people with the highest propensity to buy without presenting them with irrelevant or excessive ads, P&G Chief Executive Officer David Taylor said at a conference last year.

“We are reinventing brand building, from wasteful mass marketing to mass one-to-one brand building fueled by data and technology,” he said. “This is driving growth while delivering savings and efficiencies.”

In response to that, I tweeted,

Won’t app makers find ways to work around the no tracking ask, regardless of whether it’s a global or a one-at-a-time setting? That seems to be what the
@WSJ is saying about  @ProcterGamble ‘s work with #CAID device fingerprinting.

Don replied,

Yes. Some app developers will figure out a way to track you that doesn’t get caught by the App Store review. Apple can’t promise a complete “stop this app from tracking me” feature because sometimes it will be one of those apps that’s breaking the rules

Then Augustine Fou replied,

of course, MANY ad tech companies have been working on fingerprinting for years, as a work around to browsers (like Firefox) allowing users to delete cookies many years ago. Fingerprinting is even more pernicious because it is on server-side and out of control of user entirely

That last point is why I’ve long argued that we have a very basic problem with the client server model itself: that it all but guarantees a feudal system in which clients are serfs and site operators (and Big Tech in general) are their lords and masters. Though my original metaphor for client-server (which I have been told was originally a euphemism for slave-master) was calf-cow:

Here’s more on that one, plus some other metaphors as well:

A sense of bewronging (from 2011) Stop making cows. Quit being calves (from 2012) Is being less tasty vegetables our best strategy? (2021) Thinking outside the browser (2021) Toward e-commerce 2.0 (2021) How the cookie poisoned the Web (2021)

I’ll pick up that thread after visiting what Nick says about fingerprinting:

There are countless ways that devices can be fingerprinted, and the mandated use of IDFA instead of those surreptitious methods makes it harder for ad tech companies to be sneaky. It has long been possible to turn off IDFA or reset the identifier. If it did not exist, ad tech companies would find other ways of individual tracking without users’ knowledge, consent, or control.

And why “ask” an app not to track? Why not “tell”? Or, better yet, “Prevent Tracking By This App”? Does asking an app not to track mean it won’t?

History has an answer for those questions.

Remember Do Not Track? Invented in the dawn of tracking, back in the late ’00s, it’s still a setting in every one of our browsers. But it too is just an ask — and ignored by nearly every website on Earth.

Much like Do Not Track, App Tracking Transparency is a request — verified as much as Apple can by App Review — to avoid false certainty. Tracking is a pernicious reality of every internet-connected technology. It is ludicrous to think that any company could singlehandedly find and disable all forms of fingerprinting in all apps, or to guarantee that users will not be tracked.

I agree. This too is a problem with the feudal system that the Web + app world has become, and Nick is right to point it out. He continues,

The thing that bugs me is that Searls knows all of this. He’s Doc Searls; he has an extraordinary thirteen year history of writing about this stuff. So I am not entirely sure why he is making arguments like the ones above that, with knowledge of his understanding of this space, begin to feel disingenuous. I have been thinking about this since I read this article last night and I have not come to a satisfactory realistic conclusion.

Here’s a realistic conclusion (or at least the one that’s in my head right now): I was mistaken to assume that Apple has more control here than it really does, and it’s right for all these guys (Nick, John, Augustine, Don and others) to point that out. Hey, I gave in to wishful thinking and unconscious ad hominem argumentation. Mea bozo. I sit corrected.

He continues,

Apple is a big, giant, powerful company — but it is only one company that operates within the realities of legal and technical domains. We cannot engineer our way out of the anti-privacy ad tech mess. The only solution is regulatory. That will not guarantee that bad actors do not exist, but it could create penalties for, say, Google when it ignores users’ choices or Dr. B when it warehouses medical data for unspecified future purposes.

We’ve had the GDPR and the CCPA in enforceable forms for awhile now, and the main result, for us mere “data subjects” (GDPR) and “consumers” (CCPA) is a far worse collection of experiences in using the Web.

At this point my faith in regulation (which I celebrated, at least in the GDPR case, when it went into force) is less than zero. So is my faith in tech, within the existing system.

So I’m moving on, and working on a new approach, outside the whole feudal system, which I describe in A New Way. It’s truly new and small, but I think it can be huge: much bigger than the existing system, simply because we on the demand side will have better ways of informing supply (are you listening, Mark Pritchard?) than even the best surveillance systems can guess at.

Saturday, 29. May 2021

Doc Searls Weblog

Apple vs (or plus) Adtech, Part I

This piece has had a lot of very smart push-back (and forward, but mostly back). I respond to it in Part II, here. If you haven’t seen it yet, watch Apple’s Privacy on iPhone | tracked ad. In it a guy named Felix (that’s him, above) goes from a coffee shop to a waiting room […]

This piece has had a lot of very smart push-back (and forward, but mostly back). I respond to it in Part II, here.

If you haven’t seen it yet, watch Apple’s Privacy on iPhone | tracked ad. In it a guy named Felix (that’s him, above) goes from a coffee shop to a waiting room somewhere, accumulating a vast herd of hangers-on along the way. The herd represents trackers in his phone, all crowding his personal space while gathering private information about him. The sound track is “Mind Your Own Business,” by Delta 5. Lyrics:

Can I have a taste of your ice cream?
Can I lick the crumbs from your table?
Can I interfere in your crisis?

No, mind your own business
No, mind your own business

Can you hear those people behind me?
Looking at your feelings inside me
Listen to the distance between us

Why don’t you mind your own business?
Why don’t you mind your own business?

Can you hear those people behind me?
Looking at your feelings inside me
Listen to the distance between us

Why don’t you mind your own business?
Why don’t you mind your own business?

The ad says this when Felix checks his phone from the crowded room filled with people spying on his life:

Then this:

Finally, when he presses “Ask App Not to Track,” all the hangers-on go pop and turn to dust—

Followed by

Except that she gets popped too:

Meaning he doesn’t want any one of those trackers in his life.

The final image is the one at the top.

Here’s what’s misleading about this message: Felix would have had none of those trackers following him if he had gone into Settings—>Privacy—>Tracking, and pushed the switch to off, like I’ve done here:

Key fact: it is defaulted to on. Meaning Apple is not fully serious about privacy. If Apple was fully serious, your iPhone would be set to not allow tracking in the first place. All those trackers would come pre-vaporized. And Apple never would have given every iPhone an IDFA—ID For Advertisers—in the first place. (And never mind that they created IDFA back in 2013 partly to wean advertisers from tracking and targeting phones’ UDIDs (unique device IDs).

Defaulting the master Tracking setting to ON means Felix has to tap “Ask App Not To Track” for every single one of those hangers-on. Meaning that one click won’t vaporize all those apps at once. Just one at a time. This too is misleading as well as unserious.

And why “ask” an app not to track? Why not “tell”? Or, better yet, “Prevent Tracking By This App”? Does asking an app not to track mean it won’t?

History has an answer for those questions.

Remember Do Not Track? Invented in the dawn of tracking, back in the late ’00s, it’s still a setting in every one of our browsers. But it too is just an ask—and ignored by nearly every website on Earth.

Here is how the setting looks, buried deep on Google’s Chrome:

It’s hardly worth bothering to turn that on (it’s defaulted to off), because it became clear long ago that Do Not Track was utterly defeated by the adtech biz and its dependents in online publishing. The standard itself was morphed to meaninglessness at the W3C, where by the end (in 2019) it got re-branded “Tracking Preference Expression.” (As if any of us has a preference for tracking other than to make it not happen or go away.)

By the way, thanks to adtech’s defeat of Do Not Track in 2014, people took matters into their own hands, by installing ad and tracking blockers en masse, turning ad blocking, an option that had been laying around since 2004, into the biggest boycott in world history by 2015.

And now we have one large company, Apple, making big and (somewhat, as we see above) bold moves toward respecting personal privacy. That’s good as far as it goes. But how far is that, exactly? To see how far, here are some questions:

Will “asking” apps not to track on an iPhone actually make an app not track? How will one be able to tell? What auditing and accounting mechanisms are in place—on your phone, on the apps’ side, or at Apple?

As for people’s responses to Apple’s new setting, here are some numbers for a three-week time frame: April 26 to May 16. They come from FLURRY, a subsidiary of Verizon Media, which is an adtech company. I’ll summarize:

For “Worldwide daily op-in rate after iOS 14.5 launch across all apps,” expressed as “% of mobile active app users who allow app tracking among uses who have chosen to either allow or deny tracking” started at 11% and rose to 15%. The “U.S. Daily opt-in rate after iOS launch across all apps,” expressed as “% of mobile active app users who allow app tracking among users who have chosen to either allow or deny tracking” started at 2% and rose to 6%. The “Worldwide daily opt-in rate across apps that have displayed the prompt,” expressed as “% of mobile active app users who allow app tracking among users who have chosen to either allow or deny tracking” started at 31% and went down to 24%. The “Worldwide daily share of mobile app users with ‘restricted’ app tracking” (that’s where somebody goes into Settings—>Privacy—>Tracking and switches off “Allow Apps to Request to Track”), expressed as “% of mobile active app users who cannot be tracked by default and don’t have a choice to select a tracking option” started and stayed within a point of 5% . And the “U.S. daily share of mobile app users with ‘restricted’ app tracking,” expressed as “% of mobile active app users who cannot be tracked by default and don’t have a choice to select a tracking option” started at 4% and ended at 3%, with some dips to 2%.

Clearly tracking isn’t popular, but those first two numbers should cause concern for those who want tracking to stay unpopular. The adtech business is relentless in advocacy of tracking, constantly pitching stories about how essential tracking-based “relevant,” “personalized” and “interest-based” advertising is—for you, and for the “free” Web and Internet.

It is also essential to note that Apple does advertising as well. Here’s Benedict Evans on a slope for Apple that is slippery in several ways:

Apple has built up its own ad system on the iPhone, which records, tracks and targets users and serves them ads, but does this on the device itself rather than on the cloud, and only its own apps and services. Apple tracks lots of different aspects of your behaviour and uses that data to put you into anonymised interest-based cohorts and serve you ads that are targeted to your interests, in the App Store, Stocks and News apps. You can read Apple’s description of that here – Apple is tracking a lot of user data, but nothing leaves your phone. Your phone is tracking you, but it doesn’t tell anyone anything.

This is conceptually pretty similar to Google’s proposed FLoC, in which your Chrome web browser uses the web pages you visit to put you into anonymised interest-based cohorts without your browsing history itself leaving your device. Publishers (and hence advertisers) can ask Chrome for a cohort and serve you an appropriate ad rather than tracking and targeting you yourself. Your browser is tracking you, but it doesn’t tell anyone anything -except for that anonymous cohort.

Google, obviously, wants FLoC to be a generalised system used by third-party publishers and advertisers. At the moment, Apple runs its own cohort tracking, publishing and advertising as a sealed system. It has begun selling targeted ads inside the App Store (at precisely the moment that it crippled third party app install ads with IDFA), but it isn’t offering this tracking and targeting to anyone else. Unlike FLoC, an advertiser, web page or app can’t ask what cohort your iPhone has put you in – only Apple’s apps can do that, including the app store.

So, the obvious, cynical theory is that Apple decided to cripple third-party app install ads just at the point that it was poised to launch its own, and to weaken the broader smartphone ad model so that companies would be driven towards in-app purchase instead. (The even more cynical theory would be that Apple expects to lose a big chunk of App Store commission as a result of lawsuits and so plans to replace this with app install ads. I don’t actually believe this – amongst other things I think Apple believes it will win its Epic and Spotify cases.)

Much more interesting, though, is what happens if Apple opens up its cohort tracking and targeting, and says that apps, or Safari, can now serve anonymous, targeted, private ads without the publisher or developer knowing the targeting data. It could create an API to serve those ads in Safari and in apps, without the publisher knowing what the cohort was or even without knowing what the ad was. What if Apple offered that, and described it as a truly ‘private, personalised’ ad model, on a platform with at least 60% of US mobile traffic, and over a billion global users?…

Apple has a tendency to build up strategic assets in discrete blocks and small parts of products, and then combine them into one. It’s been planning to shift the Mac to its own silicon for close to a decade, and added biometrics to its products before adding Apple Pay and then a credit card. Now it has Apple Pay and ‘Sign in with Apple’ as new building blocks on the web, that might be combined into other things. It seems pretty obvious that Privacy is another of those building blocks, deployed step by step in lots of different places. Privacy has been good business for Apple, and advertising is a bigger business than all of those.

All of which is why I’ve lately been thinking that privacy is a losing battle on the Web. And that we need to start building a byway around the whole mess: one where demand can signal supply about exactly what it wants, rather than having demand constantly being spied on and guessed at by adtech’s creepy machinery.

Friday, 28. May 2021

Phil Windley's Technometria

Comparing X.509 Certificates with SSI

Summary: X.509 certificates have been around for 40 years and have proven to be a trustworthy means of exchanging data. So, what are the differences between X.509 ceritificates and SSI? And what are the advantages of each? I sometimes talk to people who ask "Why do we need SSI? What's wrong with X.509 certificates?" Here's some thoughts. X.509 is a standard that defines th

Summary: X.509 certificates have been around for 40 years and have proven to be a trustworthy means of exchanging data. So, what are the differences between X.509 ceritificates and SSI? And what are the advantages of each?

I sometimes talk to people who ask "Why do we need SSI? What's wrong with X.509 certificates?" Here's some thoughts.

X.509 is a standard that defines the format for public key certificates. Public key certificates can be used to tie a public key to other information. The most common use, by far, is TLS/SSL, the basis for trust in HTTPS, the protocol that secures the Web. In TLS, the certificate binds a public key to a domain name (and perhaps other information).

The first challenge for many people is determining whether X.509 certificates are more like verifiable credentials or DIDDocs. This is understandable since X.509 combines the functions of these two separate SSI standards. X.509 certificates themselves are like DIDDocs in that they bind information to a public key. But the hierarchical public key infrastructure (PKI) of X.509 is meant to attest to the veracity of the the X.509 certificate. And X.509 extensions allow other information to be included. So, X.509 certificates also bind the public key (as an identifier) to real-world attributes. DIDDocs don't have anything like PKI. Rather SSI uses verifiable credentials to assert information about a decentralized identifier in a trustworthy way.

Another important difference between X.509 certificates and DIDDocs is that the primary purpose of the DIDDoc is to bind the public key in the DIDDoc to a decentralized identifier, or DID, whereas X.509 certificates can bind the public key to a subject name and other information like a domain name. Extensions to the certificate allow it to also bind the public key to other information. The important distinction is that the DID is required and represents a unique name for the DIDDoc. A DID must have some means of resolving to the DIDDoc1. The DID provides a level of indirection to the public key. Consequently, the public key associated with a DID can be rotated without changing the DID and so it can be used as a permanent identifier. I won't get into the details around how this is done securely, but you can read far more detail at The Architecture of Identity Systems if you're curious.

The veracity of an X.509 certificate is usually determined from the strictly hierarchical public key infrastructure (PKI). For example, when you visit a web site, your browser uses the X.509 certificate from the web site to establish a secure connection. If you click on the lock, you'll see information about that certificate. The web site's certificate was signed by some organization that is attesting to the information in the certificate. You can use the certificate of the signing organization to know its public key to do the check. But how do you know that certificate is valid? It's signed using the private key whose public key is in yet another certification, and so on. Eventually this has to stop and it does when you get to a certificate that was stored in browser when it was built. CA Browser Forum is the organization that determines what certificates are worthy to be inside browsers.

Showing the Certificate Hierarchy in Brave (click to enlarge)

In contrast, the veracity of the DID and associated DIDDoc is ascertained by a heterarchical method. The DID and DIDDoc are self-asserted and self-certifying. You can use cryptographic means to determine that the binding asserted in the DIDDoc has not been tampered with, but the DID infrastructure itself does nothing to tell you who or what the DID is bound to in a verifiable way. For that, we use verifiable credentials.

Suppose the DID in question is one Alice generated to give to Bravo Corp, her mortgage processor. Bravo knows nothing about the DID they've received except that it's bound, in the associated DIDDoc, with a specific public key (and possibly an endpoint of some kind). They ask Alice to prove things about herself as part of the mortgage application process and over time learn quite a bit. Alice proves her name and date of birth using a verifiable credential representing her driver's license. She proves her income using a verifiable credential from her employer, and her banking information using a verifiable credential from her bank. The information in each of these verifiable credentials is attested by its issuer: the DMV, the employer, and the bank. Bravo's reasons for trusting these organization are up to Bravo:

The may be well known. Bravo may have a prior relationship with them. Bravo might ask them to prove things about themselves (using verifiable credentials, of course). Or they may belong to a trust framework that Bravo can use access publicly.

Furthermore, zero-knowledge proofs (ZKPs)2 allow Alice to combine the attributes in these various credentials (and others) in a way that only discloses what Bravo is asking for and nothing more. And her digital wallet was able to do this for her automatically without Alice having to pick and choose the various attributes from various credentials herself. The proof shows that the information from these three credentials is all bound to the person who controls the DID that Alice gave to Bravo. The proof also shows that these credentials have not been revoked.

You can imagine Alice having X.509 certificates from the DMV, her employer, and her bank that attest these same things (through X.509 extensions). She would also have a personal certificate with her public key that she used to anchor each of these other certificates. The X.509 certificates are not linked in any way other than Alice's public key. She has to use the same public key in all of them so they can be correlated. She uses her personal certificate to prove she's in control of the public key she provides to the DMV, employer, and bank. If she changes her public key, so has to get new certificates. This is a good example of the dual nature of X.509 certificates. Alice's personal certificate looks like a DIDDoc, but the certificates with extensions look like verifiable credentials.

There's no easy way for Alice to restrict what attributes she shares when she shares these certificates. She has to share the entire certificate. Bravo would trust these certificates in the same way your browser does, by following the chain to some smallish set of trusted certificate authorities fo each kind of certificate (driver's license, employer, or bank). Bravo would also check certificate revocation lists for each certificate to ensure they're still valid.

The advantage of X.509 certificates is that the technology, processes, and governance behind them are well-known and understood. No small thing. The public key infrastructure is well developed with a long history of securely communicating trustworthy public keys. DIDs and verifiable credentials are relatively new. Although standards, open source code, and multiple vendors exist, they are unproven compared to X.509.

So, why do something new? DIDs, DIDDocs, and verifiable credentials have several advantages over X.509 certificates:

DIDs are more secure. DIDs allow public keys to be rotated in a trustworthy manner. Consequently, Alice can rotate the key underlying the DID at will without having to get new credentials. The identifier lives as long as Alice needs it to. Alice won't be tempted to hold onto a potentially comprimised key because she's worried about the inconvenience. SSI uses the right tools for each part of the process. The SSI architecture cleanly separates providing an identifier for Alice from proving things about Alice. The binding between the DID and its associated public key can be verified cryptographically without relying on a hierarchical chain of authorities. The fidelity of the credential exchange can be verified cryptographically using information in a public credential registry (often a ledger of some sort). This separation allows the methods and tools to be crafted to the needs of each kind of document. Verifiable credentials minimize information disclosure. Sharing only what's necessary protects Alice's privacy. This Webinar on ZKP-oriented Credentials from Daniel Hardman is an excellent, approachable tutorial on the many benefits of ZKPs for credential exchange. SSI data sharing UX is safer. ZKPs provide convenience for Alice saving her time, and reducing the chance of her oversharing through human error (i.e. they are safer from a privacy perspective). SSI has a consistent UX. SSI wallets and agents provide a good user experience for managing relationships, storing credentials, and responding to proof requests. As far as I know, X.509 certificate wallets do not exist as such, so they would need to be developed to provide a comparable user experience. Verifiable credentials provide better interoperability. Alice is able to use multiple credentials from different issuers and prove things to many verifiers because of standards, not just for data formats, but also protocols for issuace and presentment. I know of no standards for how X.509 credentials can be used to prove the kind of information in the mortgage example in an interoperable way. They have been around for over 40 years and yet they are almost exclusively used for TLS and nothing else.

The high-level goals of X.509 certificates are similar to those of DIDs and verifiable credentials. But DIDs and verifiable credentials represent an innovation that takes learnings from 40 years of experience and new developments in cryptography into account to provide a better, more flexible solution to the problem of exchanging data in a trustworthy way. SSI in the form of DIDs and verifiable credentials promise a global, interoperable data exchange metasystem that is cryptographically sound with an excellent user experience.

Notes The resolution need not be global or public. For Peer DIDs, the resolution is local. Note that not all credential exchange methods use ZKPs. They should.

Tags: ssi identity x.509 pki decentralized+identifiers

Wednesday, 26. May 2021

FACILELOGIN

Introducing Keto, the open source implementation of Google’s Zanzibar

https://deih43ym53wif.cloudfront.net/zanzibar-tanzania-2_7e6a147e43.jpeg Zanzibar is Google’s consistent, global authorization system; a project started at Google in 2012. It primarily plays two roles. It acts as a storage system for access control lists and groups used by hundreds of Google projects (Maps, Google Drive, Calendar, and so on) to store permissions of digital objects. Also, Zanzibar a
https://deih43ym53wif.cloudfront.net/zanzibar-tanzania-2_7e6a147e43.jpeg

Zanzibar is Google’s consistent, global authorization system; a project started at Google in 2012. It primarily plays two roles. It acts as a storage system for access control lists and groups used by hundreds of Google projects (Maps, Google Drive, Calendar, and so on) to store permissions of digital objects. Also, Zanzibar acts as an authorization engine, where the Google products talk to check whether a given request is authorized or not.

At the moment Zanzibar manages 2 trillion+ tuples of 1500+ namespaces, replicated in several dozens of locations globally over 10,000+ servers. It handles 10million+ client queries per second.

I recently learnt about Keto, an open source implementation of Zanzibar from a Hacker News thread, and thought it would be a very interesting topic to discuss in our Silicon Valley IAM Meetup and invited Patrik Neu, who is the core maintainer of the Keto project to share his insights with our meetup community. Patrik is based out of Munich, Germany and he’s been contributing to open source projects since 2017.

The Ory, is the company behind Keto, and they are also in the process of building a SaaS offering on top of Keto.

Introducing Keto, the open source implementation of Google’s Zanzibar was originally published in FACILELOGIN on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tuesday, 25. May 2021

Mike Jones: self-issued

OpenID Connect Federation updated in preparation for third Implementer’s Draft review

The OpenID Connect Federation specification has been updated to add Security Considerations text. As discussed in the recent OpenID Connect working group calls, we are currently reviewing the specification in preparation for it becoming the third and possibly last Implementer’s Draft. Working group members (and others!) are encouraged to provide feedback on the draft soon […]

The OpenID Connect Federation specification has been updated to add Security Considerations text. As discussed in the recent OpenID Connect working group calls, we are currently reviewing the specification in preparation for it becoming the third and possibly last Implementer’s Draft.

Working group members (and others!) are encouraged to provide feedback on the draft soon before we start the foundation-wide review. We will probably decide next week to progress the draft to foundation-wide review. In particular, there’s been interest recently in both Entity Statements and Automatic Registration among those working on Self-Issued OpenID Provider extensions. Reviews of those features would be particularly welcome.

The updated specification is published at:

https://openid.net/specs/openid-connect-federation-1_0-16.html

Special thanks to Roland Hedberg for the updates!


Phil Windley's Technometria

Life-Like Anonymity and the Poison Web

Summary: Natural anonymity comes from our ability to recognize others without the aid of an external identity system. Online interactions will only be able to mirror life-like anonymity when we can use decentralized identity systems that don't force all interactions to be under the purview of centralized, administrative identity systems. Doc Searls published a piece last week entitl

Summary: Natural anonymity comes from our ability to recognize others without the aid of an external identity system. Online interactions will only be able to mirror life-like anonymity when we can use decentralized identity systems that don't force all interactions to be under the purview of centralized, administrative identity systems.

Doc Searls published a piece last week entitled "How the Cookie Poisoned the Web". Doc points to various privacy ills of Web 2.0 and in each instance says "Blame the cookie." Doc's larger point is that the web started out as a peer-to-peer publishing system that was wholly decentralized and gave everyone equal voice.

Doc continues:

But gradually a poison disabled personal agency. That poison was the cookie.

Very few web sites in the early web had identity systems. For the peer-to-peer sharing of documents and discovery via embedded links, none were needed. HTTP, the foundational protocol of the web is stateless, meaning the HTTP server does not know whether any two requests are related to each other.

Stateless is fine for document sharing and linking using hypertext. But it makes building a shopping cart really hard. Back in the mid-90's figuring out how to build a functional shopping cart was on everyone's mind, mine included. I was the cofounder and CTO of an early ecommerce site, imall.com. Without changing HTTP, the most promising strategy was to include a correlation identifier in all the links generated by the site, so we'd know who was making the request. But this was buggy and caused lots of customer support issues.

A correlation identifier is a unique string that can be used to link requests. Ultimately, the the HTTP community added a correlation identifier called a "cookie" (which took its name from a correlation identifier used in unix). HTTP cookies are generated by the server and stored on the browser. Whenever the browser makes a request to the server, it sends back the cookie, allowing the server to correlate all requests from that browser.

That all sounds innocuous enough and in theory, it is. But the devil is in the details. If I'm shopping on imall.com, I want the site to keep track of me because that provides utility and convenience. But it turns out that most web pages are not a single chunk of HTML that the server sends down. They have lots of other things, like javascript files and images, embedded in them too. These other things don't have to be from the same server. Each of those servers can set a cookie as well. And since they know where they were linked from, they can correlate activity across multiple websites.

This is how (simple) ad tracking works. When you see an ad on web site A, it's being served from a server owned by an ad company that web site A has an agreement with. The ad server plants a cookie in your browser. Now you visit web site B that also includes ads from the same ad server. Your browser dutifully reports the ad server cookie back to the ad server along with the information that the ad was on web site B. The company running the ad server now knows you were on web site A and web site B (along with lots of other metadata). Rather than correlating requests on a single web site, they are using cookies to correlate your activity across the web.

This is the poison Doc is talking about. The web cookie, as designed, goes well beyond correlating activity on a single web site for purposes of creating some utility like a shopping cart or a chat server. The web cookie allows correlating activities of people across the web. And it doesn't stop with your browsing history. The ad company starts knowing other things about you (because the web sites you visit tell them) and soon they can develop a comprehensive dossier.

Like-Like Anonymity and the Administrative Internet

In real life, we often interact with others—both people and institutions—with relative anonymity. For example, if I go the store and buy a coke with cash there is no exchange of identity information necessary. Even if I use a credit card it's rarely the case that the entire transaction happens under the administrative authority of the identity system inherent in the credit card. Only the financial part of the transaction takes place in that identity system. This is true of most interactions in real life.

In contrast, in the digital world, very few meaningful transactions are done outside of some administrative identity system. There are several reasons why identity is so important in the digital world:

Continuity—While web sessions can be pseudonymous, as we've seen, they are often correlated across multiple independent sessions and devices using an authenticated correlation identifier. This allows, for example, the customer to have a shopping cart that not only persists across time but also on different devices. Convenience—So long as the customer is authenticating, we might as well further store additional information like addresses and credit card numbers for their convenience, to extend the shopping example. Storing these allows the customer to complete transactions without having to enter the same information over and over. Trust—There are some actions that should only be taken by certain people, or people in certain roles, or with specific attributes. Once a shopping site has stored my credit card, for example, I ought to be the only one who can use it. Identity systems provide authentication mechanisms as the means of knowing who is at the other end of the wire so that we know what actions they're allowed to take. This places identifiers in context so they can be trusted. Surveillance—Unfortunately, identity systems also provide the means of tracking individuals across transactions for purposes of gathering data about them. This data gathering may be innocuous or nefarious, but there is no doubt that it is enabled by identity systems in use on the internet.

In real life, we do without identity systems for most things. You don't have to identify yourself to the movie theater to watch a movie or log into some system to sit in a restaurant and have a private conversation with friends. In real life, we act as embodied, independent agents. Our physical presence and the laws of physics have a lot to do with our ability to function with workable anonymity across many domains.

So, how did we get surveillance and it's attendant affects on natural anonymity as an unintended, but oft-exploited feature of administrative digital identity systems? Precisely because they are administrative.

Legibility

Legibility is a term used to describe how administrative systems make things governable by simplifying, inventorying, and rationalizing things around them. James C. Scott's seminal book, Seeing Like a State, nicely analyzes legibility and its unintended consequences. Venkatesh Rao has a great summary if you'd like the TL;DR.

Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed by James C. Scott

In this wide-ranging and original book, James C. Scott analyzes failed cases of large-scale authoritarian plans in a variety of fields. Centrally managed social plans misfire, Scott argues, when they impose schematic visions that do violence to complex interdependencies that are not—and cannot—be fully understood. Further, the success of designs for social organization depends upon the recognition that local, practical knowledge is as important as formal, epistemic knowledge.

Identity systems make people legible in order to offer continuity, convenience, and trust. But, as we've seen, that legibility also allows surveillance. In some respects, this is the trade off we always get with administrative systems. By creating legibility, administrative systems threaten privacy.

Administrative systems are centralized. They are owned. They are run for the purposes of their owners, not the purposes of the people or things being administered. They are bureaucracies for governing something. They rely on rules, procedures, and formal interaction patterns. Need a new password? Be sure to follow the password rules of what ever administrative system you're in.

Every interaction you have online happens under the watchful eye of a bureaucracy built to govern the system and the people using it. The bureaucracy may be benevolent, benign, or malevolent but it controls the interaction and people pay the price of the interpretive work necessary to figure out how it functions.

Real Life is Decentralized

On the other hand, in real life we interact as peers. We do interact with administrative systems of various sorts, but no one would describe that as real life. When I go to a store, I don't think about shopping within their administrative system. Rather, I walk in, look at stuff, talk to people, put things in a cart, and check out. The administrative system is there, but it's for governing the store, not the customers.

We can't have online interactions that feel like real life until we redecentralize the internet. The internet started out decentralized. The early web was decentralized. But the need for continuity, convenience, and trust led more and more interactions to happen within someone's administrative system.

Most online administrative systems make themselves as unobtrusive as they can. But there's no getting around the fact that every move we make is within a system that knows who we are and monitors what we're doing. In real life, I don't rely on the administrative system of the restaurant to identify the people I'm having dinner with. The restaurant doesn't need to check our IDs or surveil us in order to create an environment where we can talk and enjoy a meal together.

The good news is that we're finally developing the tools necessary to create decentralized online experiences. What if you could interact with your friends online on the basis of an identity that they bring to you directly—one that you could recognize and trust? You wouldn't need Facebook or WhatsApp to identify and track your friends for you.

Decentralized identity is the foundation for a decentralized web—a web that flexibly supports the kind of ad hoc interactions people have with each other all the time in real life. Until we do, we'll never get an online world that mirrors real life and its natural anonymity.

Photo Credit: Poison Apple Sugar Cookies from Angelica Made Me (unknown)

Tags: identity web surveillance+capitalism decentralization


MyDigitalFootprint

Do our tools mould our outcomes and decisions?

Day zero of being a #CDO is probably not the best day to ask difficult questions; however, sometimes, there is no better day.  The first question to ask the executive leadership team as you walk around being introduced might be: “What is the one thing that we, as a team and organisation, want our data to drive, deliver or provide?” You might want to wait to ask this question and first de

Day zero of being a #CDO is probably not the best day to ask difficult questions; however, sometimes, there is no better day.  The first question to ask the executive leadership team as you walk around being introduced might be: “What is the one thing that we, as a team and organisation, want our data to drive, deliver or provide?”

You might want to wait to ask this question and first determine what tools are being used. This will frame what outcomes and decisions are being supported.  The question and answers allow you to determine if there is an alignment or gap between “What is the one thing that we, as a team and organisation what our data to drive, deliver or provide?”  and what will happen anyway because of tools, processes and legacy.  One critical aspect of being a #CDO is determining how our processes and methods only enable certain decisions to be made, but we have to unpack legacy.

Legacy within this framing is threefold. Decisions. Decisions. Decisions. These are:  

Decisions that created processes, methods and rules which were created by previous incentives and power games; and are now ghosts in the systems. These decisions were taken so long ago that no one knows why, how or when it was decided.  It is the way we do it; it is our IP, our brand.

Decisions that created “information and technology debt” included embedded and baked-in systems, hidden, and no-longer supported code, and automation based on tools and data biased when created. 

Decisions that created noise in the hierarchy to lose or filter signals that someone did not want to hear.  It was the creation of layers, reports, practices, structural regulation and unchallenged assumptions.

Unpacking legacy questions will take time, and you will not be ready on Day-0. Therefore any responses you get have to be framed within such understanding when it arrives.  It is worth asking the question and then verifying before becoming blind to the tools that mould you. 

“What is the one thing that we, as a team and organisation what our data to drive, deliver or provide?  Well, it could be:

Evidence-based, actionable insights

What should we automate?

How do we know we are doing the right thing?

Where are there efficiencies to be gained?

What do customers really want?

How to manipulate customers to increase margin and revenues?

Where are risks that we cannot see?

What is being hidden that we cannot see?

If you look at this list in the context of what tools and decisions already frame the response, are these what we are looking for data to answer or are we looking to data to affirm/ justify what we have already decided.  A different response that no one will say “to justify what we are already doing!”

Data has bias because of previous decisions. Therefore, a #CDO has to find non-data tools to check what decisions from the past are biasing the current data, processes and tools. We cannot usefully answer the question we have set ourselves “What is the one thing that we, as a team and organisation what our data to drive, deliver or provide? without understanding the situation.

The CTO knows that they have to build a new platform when the bug list, new feature development and maintenance costs are bigger and will take more time than developing a new platform - this is the technology debt question.  As the #CDO, you have to understand what is your information debt. The CTO will struggle as there is no clear path from policy to code. Similarly, the CDO, we struggle with no clear path from policy (what one thing) to better data for the decisions we require.  You inherit and are now accountable for previous decisions, which constrain what is now possible as the biased tool has created what we have. The costs of collecting, labelling, holding, sorting and creating training data continually increase, creating a more significant gap and misalignment in values and expectations from data.

“We become what we behold. We shape our tools, and then our tools shape us”  is often mistakenly attributed to Marshall McLuhan (as in the image above) and called McLuhan Law. The quote was actually written by Father John Culkin, SJ, a Professor of Communication at Fordham University in New York and friend of McLuhan. Such is the problem with data. 

Perhaps the opening question is the wrong one? Perhaps we should reflect on these questions as a leadership team.

What do we want to become, and what tools and data will help us?

What tools do we use, and what will they enable us to become? 

What is the minimum viable data set required to give the best value? 

Do our tools and data trap us?



Hans Zandbelt

Using an OAuth 2.0 Resource Server with Certificate-Bound Access Tokens

ZmartZone has implemented OAuth 2.0 Resource Server functionality in Apache/NGINX modules so these components can be used as a reverse proxy in front of APIs or other backends. In such a setup the backend does not have to deal with … Continue reading →

ZmartZone has implemented OAuth 2.0 Resource Server functionality in Apache/NGINX modules so these components can be used as a reverse proxy in front of APIs or other backends. In such a setup the backend does not have to deal with security but outsources it to a proxy sitting in front of it in a similar way that TLS termination is often offloaded to a load-balancer.

Most of the OAuth 2.0 deployments today use so-called bearer access tokens that are easy to deploy and use. This type of access token is not bound to the Client presenting the token which means that an attacker intercepting an access token can just use that token to get access to the resources/APIs/services as if it were the Client. So called Proof-of-Possession semantics for access tokens prevent that type of attack and present a more secure setup but such a system is typically harder to implement, deploy and maintain.

A relative simple variant of Proof-of-Possession for access tokens is specified in RFC 8705 OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens. This specification leverages a (possibly self-signed) certificate held by the Client to bind an access token cryptographically to the private key associated with that certificate.

This specification is implemented in liboauth2 1.4.1 which is used in the Apache module mod_oauth2 3.2.1. This means that you can now require and verify OAuth 2.0 certificate bound access tokens for your API in a very simple way that is easy to deploy. All it takes is an Apache server in front of your API, configured with something like:

AuthType oauth2
OAuth2TokenVerify jwk "{\"kty\":\"RSA\",\"kid\":\"one\",\"use\":\"sig\",\"n\":\"...\",\"e\":\"AQAB\" }" type=mtls&mtls.policy=optional
SSLVerifyClient optional_no_ca

Acknowledgement: this work was performed in a partnership with Connect2ID. Thanks Vladimir Dzhuvinov.

Monday, 24. May 2021

MyDigitalFootprint

What occurs when physical beings transition to information beings?

You are standing in front of Deep Thought, as in the AI in HitchHicker Guide to the Galaxy.  You are tasked with asking the famous question.  “What is the answer to the world, the universe and everything?”  In Douglas Adams original work, Deep Thought responds by saying, “it will take some time”; before coming back sometime later with the answer “42.”  However, this is when

You are standing in front of Deep Thought, as in the AI in HitchHicker Guide to the Galaxy.  You are tasked with asking the famous question.  “What is the answer to the world, the universe and everything?”  In Douglas Adams original work, Deep Thought responds by saying, “it will take some time”; before coming back sometime later with the answer “42.”  However, this is when data was a statistic, and we were “physical beings” who had not considered what experience would emerge as data became dynamic and we transitioned to “information beings.”

In our new context as informational beings, Deep Thought would not just take our question, go off to think about it but would respond.  Deep Thought would do what we do and ask for clarification and to check the understanding of the question. I can imagine Deep Thought asking, “I just wanta check what you mean by ‘the answer to the world the universe and everything’ and do you want a brave answer, comfortable, or courageous answer?  I can do a kind, generous or safe one too? What type of answer do you want?”

This is the problem we are facing.  The questions that we have learnt for governance and oversight over millennia were fabulous to determine who decides, and who decides who decides in a physical world. It was where a human was held responsible and accountable.  The issue of oversight becomes very differnt when we have automation and machines/ AI make decisions for us, how do we know who decides?  Our questions will not cut it.  

When we apply our old governance, old oversight, old accountable models and old frameworks to new problems, it feels like there is a disconnect and there is a level of dysfunctionality as the old model was not designed for the new problems.  There is a forward capability; insomuch that old problems can be addressed better on the new systems, but the old systems cannot help solve or answer new questions.  There is a backwards capability issue (not backwards compatibility), just like you cannot run the latest software on the early computers. 

We appear to be right now lost (which the media portrays as division or civilisation collapse creating divides to gain clicks and generate revenue). In reality, we are transitioning from "Physical"  to "Information" beings.  I take comfort that the best minds at MIT cannot decide either:  Atoms to Bits OR  Bits to Atoms. We know if MIT is wrestling, then we are in a transition.

The transition to information beings is reaching a chasm or hurdle, as the old model, which we depended on for governance and oversight, is at the edge of its limits.  Data can perfectly tell us how effective and efficient everything is, including the deviance to plan, but data could not tell us if the plan was right or if we are doing the right thing, efficacy.  We depended on the oversight model of humans for this.  The former, efficiency, is easy, the latter being really quite tricky as it is so contextual to the moment in time and values/ principles we currently hold.  Data can tell us from where we came, our models based on said data can suggest where we will go, but as yet, we cannot determine if we should. As an adventurer, I tend to do it anyway, ignoring should, because it is there, and my scars are evidence that it does not always work out well. 

The data, models, algorithms, ML and AI we have now created are capable of informing us if it is the right thing to do, but how do we know?

Writing things down, TV, Radio, Social media, books, screens, and printing created threats to established power structures that the old system of control adapted to embrace the new technology over time. However, I believe that the current oversight, governance, compliance and regulatory controls, which are based on the last 5000 years of adaption as physical beings, which are not bad in truth, CANNOT adapt to the next evolution from physical to information. Indeed our infrastructure gave rise to data, which is creating us as information beings.  Our current models of helping us determine if we are doing the right thing fall apart as we evolve to become information beings because of a shift in power and control. Our current oversight was not designed for autonomous decision-making by a non-human, and whilst the old methods have adopted too many situations, this move to information beings appears, right now, to break the capabilities of current governance. 

We feel the gap when we read the news but don’t as yet have the language or words to describe it.   We deny and hide from this reality by focussing on back to normal growth and trying to create new models.  

Governance follows change; it does not lead to a change. Governance and oversight help set up stability when we arrive at a new place. It is a flywheel for path dependency until someone breaks the axel. We are lucky to be alive now, seeing and witnessing this transition and can apply our minds to support this.   

as physical beings over generations, we learn that 

power and control matter

monopoly and scale in a vertical matter

accountable and responsible limited to the very narrow works, and we can ignore any wider consequences 

relationships have limited context

values and principles change slowly, allowing for rules to create an impression of stability

governance and oversight work to determine if we are doing the right thing by slowing decision time, filtering noise and hierarchy to create signals


as informational beings, we have not quite worked out the new game, however;

information and data must have attestation and rights as they matter

real-time learning models with computational power really matters

dependent and inter-dependence ecosystems ensure consequences matter

commons and the quality of the human experience on earth matter

collective and individual accountability and responsibility enable oversight in real-time


I believe we feel lost and vulnerable as we are starting to see that our lack of diversity, biased education, retention of power and control mechanisms, along with oversight and governance, worked for some of us some of the time as physical beings.  However, they are not suitable when we arrive at "information beings," wanting everything to work for most people most of the time. Utopia is still a dream.  We are at the start of a journey, and right now, we lack language, words, stories, rhetoric, meaning, models, structures, frameworks and a north star. 

We know that growth fuels increasing salaries which is a mechanism for control by creating more debt, as you can borrow more.  Growth hides and maintains the lie about change for longer. It is the flywheel. Increasing lifestyles create more people who have to remain aligned,  the balance of sufficient salary to have enough debt that you remain conformist and will do as needed.  Data means we can now pick up signals from employees about who is a conformist because of debt/ trapped.  They are the easiest to corrupt, will be the weak link in the security chain, allows someone to control or exert undue influence - keep the model and the belief.  The wheels are coming off, and the axels are breaking.  

What happens when AI (who is not paid) and is not aligned to the power plays makes decisions for the best outcome.  How will you control the AI, like we use salary for humans?

Such interesting times



The diminishing value of a data set

Source:  worth observing that this Dilbert was 1993 !!!!  Radar and sonar are incredible inventions as they allow us to perceive what cannot be actually seen with the naked eye.  As technology has advance and big data analysis has emerged we have gone from a simple echo to high-quality resolution.  However, the peak value for Radar is that it informs you something is there

Source:  worth observing that this Dilbert was 1993 !!!! 

Radar and sonar are incredible inventions as they allow us to perceive what cannot be actually seen with the naked eye.  As technology has advance and big data analysis has emerged we have gone from a simple echo to high-quality resolution.  However, the peak value for Radar is that it informs you something is there which requires low resolution and very little data.  As Radar resolution has improved we can get direction and speed which requires a little more time. This new information definitely adds value to any required reactive decision. The identification of what the actual object is through increased resolution has an incremental value but not as much as knowing it is there and what direction at what speed but such information can lead to a better decision but suddenly there is an economics of cost compared to the incremental improvement in outcome.  Knowing what type of bird by species or what plane by manufacturer, does add cost and size of data set but it does not add any value in terms of the decision requirement. 

Today, current data scientists, enthusiasts and idealists are asking you to store vast quantities of data from your customer and eco-system in the hope that at some point the algorithms and AI technology will give you new rich and deep insights, such as the equivalent resolution of what rivet types are used in construction for the inbound missile for Radar. We have to question when even though technically possible,  does the resolution stop adding value?

The more data you collect the better the decisions you will be able to make in the future is a lie that is driving boards to take on the cost for data and data risk that is beyond their scope of any calculatable return.  “Collect all the data and store it as you don’t know what we might discover” is hope and choice, it is not a rational business decision. The hope is wrapped up in a power pitch that our competitive advantage will be the discovery and ability to perform magical acts of mind manipulation which will supercharge sales through the control of customers. Or is it fear that everyone else will have this and we will have no future, FOMO of the big data promise. 

Investing in the future is a fantastic idea when collecting moon or martian dust and storing it until we improve our science tools and instruments but for a company right now this is a lie.   We are collecting and storing data in the hope that an algorithm will be able to mine more value than the cost of collecting, storing and algorithm development.  Right now there is little evidence from all the work done that using data to change behaviour works at scale.  This is a real cost today for “hope,” which the CFO has not found an ethical way to put on a balance sheet.  

We have been sold a vision the more data we have the better decisions and insights we will have. Any data scientist or statistician will confirm that resolution of a decision or insight will not substantially improve beyond a certain data set size.  You are not keeping and storing data for better decision making and insight today, just in the hope for new insights tomorrow. But the COO, CIO, CDO or CTO who is empire-building and determine that size equates to power within your culture are dependent on fear, uncertainty and doubt and insufficiently skilled directors to be able to challenge this. The tech looks like magic and every consulting company sees this as a revenue stream and write reports to support this view.

As a sceptic of the big data story, this article nicely aligns with your affinity and belief.  You see that your next action is the stop the ever-increasing data spend and focus on traditional business.  Whilst difficult to understand data is the right answer to business problems.  The takeaway should be to question: Where is the diminishing return for data for your organisation?  How do you know the data is good? How do you know the analysis is good. How do you know if what you are doing is the right thing to do? How do you incentive the right actions?

Data is not the problem, too much data will not create the return we hope for based on the cost we have, but we have to remain focussed that data is essential. How do we make data work for most of the people most of the time and not a few people on the odd occasion. 


Damien Bod

Secure an Angular SPA and an ASP.NET Core API using Auth0

This article shows how to implement an Angular single page application with an ASP.NET Core API and secured using the Open ID Connect code flow with PKCE and OAuth JWT Bearer tokens to protect the API. The identity provider is implemented using Auth0. The flow uses refresh tokens to renew the SPA session and the […]

This article shows how to implement an Angular single page application with an ASP.NET Core API and secured using the Open ID Connect code flow with PKCE and OAuth JWT Bearer tokens to protect the API. The identity provider is implemented using Auth0. The flow uses refresh tokens to renew the SPA session and the revocation endpoint is used to clean up the refresh tokens on logout.

Code: https://github.com/damienbod/Auth0AngularAspNetCoreApi

Setup

The solutions consists of three parts, an ASP.NET Core API which would provide the data in a secure way, an Angular application which would use the data and the Auth0 service which is used as the identity provider. Both applications are registered in Auth0 and the refresh tokens are configured for the SPA. The API can be used from the SPA application.

Angular SPA Code flow PKCE with refresh tokens

The Angular Open ID Connect client is implemented using the npm package angular-auth-oidc-client. The Auth0 client requires two special configurations to use an API. The audience is added as a custom parameter in the authorize request so that the required API can be used. The customParamsRefreshToken is used to add the scope parameter to the refresh request which is required by Auth0. The rest is standard Open ID Connect settings used for code flow using PKCE and refresh tokens.

import { APP_INITIALIZER, NgModule } from '@angular/core'; import { AuthModule, LogLevel, OidcConfigService } from 'angular-auth-oidc-client'; export function configureAuth(oidcConfigService: OidcConfigService) { return () => oidcConfigService.withConfig({ stsServer: 'https://dev-damienbod.eu.auth0.com', redirectUrl: window.location.origin, postLogoutRedirectUri: window.location.origin, clientId: 'Ujh5oSBAFr1BuilgkZPcMWEgnuREgrwU', scope: 'openid profile offline_access auth0-user-api-spa', responseType: 'code', silentRenew: true, useRefreshToken: true, logLevel: LogLevel.Debug, customParams: { audience: 'https://auth0-api-spa', // API app in Auth0 }, customParamsRefreshToken: { scope: 'openid profile offline_access auth0-user-api-spa', }, }); } @NgModule({ imports: [AuthModule.forRoot()], providers: [ OidcConfigService, { provide: APP_INITIALIZER, useFactory: configureAuth, deps: [OidcConfigService], multi: true, }, ], exports: [AuthModule], }) export class AuthConfigModule {}

An AuthInterceptor class is used to add the access token to the API requests to the secure APIs which use the access token. It is important that the access token is only sent to the intended API and not every outgoing HTTP request.

import { HttpInterceptor, HttpRequest, HttpHandler } from '@angular/common/http'; import { Injectable } from '@angular/core'; import { AuthService } from './auth.service'; @Injectable() export class AuthInterceptor implements HttpInterceptor { private secureRoutes = ['https://localhost:44390']; constructor(private authService: AuthService) {} intercept( request: HttpRequest<any>, next: HttpHandler ) { if (!this.secureRoutes.find((x) => request.url.startsWith(x))) { return next.handle(request); } const token = this.authService.token; if (!token) { return next.handle(request); } request = request.clone({ headers: request.headers.set('Authorization', 'Bearer ' + token), }); return next.handle(request); } }

ASP.NET Core API OAuth

The ASP.NET Core API allows requests from the calling SPA application. CORS is enabled for the application. The AddAuthentication method is used to add JWT bearer token security and the policies are added to verify the access token. The UseAuthentication method is used to add the security middleware.

public void ConfigureServices(IServiceCollection services) { // ... JwtSecurityTokenHandler.DefaultInboundClaimTypeMap.Clear(); // IdentityModelEventSource.ShowPII = true; // only needed for browser clients services.AddCors(options => { options.AddPolicy("AllowAllOrigins", builder => { builder .AllowCredentials() .WithOrigins( "https://localhost:4204") .SetIsOriginAllowedToAllowWildcardSubdomains() .AllowAnyHeader() .AllowAnyMethod(); }); }); services.AddAuthentication(options => { options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme; options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme; }).AddJwtBearer(options => { options.Authority = "https://dev-damienbod.eu.auth0.com/"; options.Audience = "https://auth0-api-spa"; }); services.AddSingleton<IAuthorizationHandler, UserApiScopeHandler>(); services.AddAuthorization(policies => { policies.AddPolicy("p-user-api-auth0", p => { p.Requirements.Add(new UserApiScopeHandlerRequirement()); // Validate id of application for which the token was created p.RequireClaim("azp", "Ujh5oSBAFr1BuilgkZPcMWEgnuREgrwU"); }); }); services.AddControllers(options => { var policy = new AuthorizationPolicyBuilder() .RequireAuthenticatedUser() .Build(); options.Filters.Add(new AuthorizeFilter(policy)); }); } public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { // ... app.UseCors("AllowAllOrigins"); app.UseHttpsRedirection(); app.UseRouting(); app.UseAuthentication(); app.UseAuthorization(); app.UseEndpoints(endpoints => { endpoints.MapControllers(); }); }

The UserApiScopeHandler class implements the AuthorizationHandler to require the UserApiScopeHandlerRequirement requirement which is used as the policy.

public class UserApiScopeHandler : AuthorizationHandler<UserApiScopeHandlerRequirement> { protected override Task HandleRequirementAsync(AuthorizationHandlerContext context, UserApiScopeHandlerRequirement requirement) { if (context == null) throw new ArgumentNullException(nameof(context)); if (requirement == null) throw new ArgumentNullException(nameof(requirement)); var scopeClaim = context.User.Claims.FirstOrDefault(t => t.Type == "scope"); if (scopeClaim != null) { var scopes = scopeClaim.Value.Split(" ", StringSplitOptions.RemoveEmptyEntries); if (scopes.Any(t => t == "auth0-user-api-spa")) { context.Succeed(requirement); } } return Task.CompletedTask; } }

The UserOneController class uses the policy which validates the access token and the claims from the token.

[SwaggerTag("User access token protected using Auth0")] [Authorize(Policy = "p-user-api-auth0")] [ApiController] [Route("api/[controller]")] public class UserOneController : ControllerBase { /// <summary> /// returns data id the correct Auth0 access token is used. /// </summary> /// <returns>protected data</returns> [HttpGet] [ProducesResponseType(StatusCodes.Status200OK)] [ProducesResponseType(StatusCodes.Status401Unauthorized)] public IEnumerable<string> Get() { return new List<string> { "user one data" }; } }

Problems, notes, Improvements

Auth0 supports the revocation endpoint which is really good and so the refresh token can be revoked when the Angular application is logged out. This is really a MUST I think if using refresh tokens in the browser. It is not possible to revoke the access tokens so these remain valid after the SPA app logs out. You could reduce the lifespan of the access tokens which would improve this a bit. Auth0 does not support reference tokens and introspection which I would always use if using SPA authentication. Introspection could be supported by using a different identity provider. Using refresh token rotation is really important when using refresh tokens in the browser, this should also be configured.

Using Auth0 with an SPA means you cannot fully logout. The tokens are also stored somewhere in the browser, but at least the refresh token can be revoked which is really important. To improve security, you could switch to a BFF architecture and remove the tokens from the browser. Then it would also be possible to fully logout. The BFF also allows for client authentication and other security features which are not possible with an SPA.

Links

https://auth0.com/docs/quickstart/webapp/aspnet-core

https://docs.microsoft.com/en-us/aspnet/core/security/authorization/introduction

Open ID Connect

https://www.npmjs.com/package/angular-auth-oidc-client

Friday, 21. May 2021

Hyperonomy Digital Identity Lab

NETAGO Downtime – May 20-21, 2021

Hardwired ThinkPad laptop Edition: Windows 10 Pro Version: 20H2 OS build: 19042.928 716 failures 75.67% downtime WiFi Wireless Lenovo laptop Edition: Windows 10 Pro Version: 20H2 OS build: 19042.985 753 failures 6.86% downtime
Hardwired ThinkPad laptop Edition: Windows 10 Pro Version: 20H2 OS build: 19042.928 716 failures 75.67% downtime

WiFi Wireless Lenovo laptop Edition: Windows 10 Pro Version: 20H2 OS build: 19042.985 753 failures 6.86% downtime netuptime-20210519-223828-tp-hardwiredDownload netuptime-20210518-051106-lenovo-wifiDownload

Thursday, 20. May 2021

MyDigitalFootprint

What Happens When the (commercial) Model Wins?

source : https://dilbert.com/strip/2021-05-20 The Insurance model wins through fear which makes you play it safe and limit risk; as they only payout for sure bets. The Banking model wins when growth fuels a salary increase and you borrow more than you have or save.  The Food production model wins when you stop cooking and you become addicted to sugar and salt (convenience) as the
source : https://dilbert.com/strip/2021-05-20

The Insurance model wins through fear which makes you play it safe and limit risk; as they only payout for sure bets.

The Banking model wins when growth fuels a salary increase and you borrow more than you have or save. 

The Food production model wins when you stop cooking and you become addicted to sugar and salt (convenience) as they control all the supply.

The Supplements model wins by fear and misinformation because you are not cooking. 

The Logistics and transport model wins with long complex supply chains due to higher levels of specialism creating better EOS.

The Legal market wins every way no matter what anyone else does, but especially the wealthy.

The Pharma Industry wins when you medicate and hide the issue rather than solve root causes.

The Consulting market wins because they promote that someone else is winning which drives the fear of losing your job because the banking industry has control by indebtedness of you both.

The Compute/ tech/ IT model wins by remaining sufficiently advanced to appear like magic.

All the Extraction models win by creating demand for new and throw away of the old to recycle, which only reduces their extraction costs.

The Waste model wins by media, Instagram, images, lifestyle and no one incentivized to recycle or repair.

The Energy model wins by their efficiency works, and everyone's else's fails or being too lazy.

The Media industry win by division, snippets, shortness, news and fueling opinion.

The Travel and entertainment market wins because it enables us to avoid for a moment that we are losing to the models that control us.

The Economic and money model wins because we all believe it.

The Species model wins by growth, and our interdependent economic models work together without having to worry if it is doing the right thing. 

Ecology wins and loses together but has an ebb and flow of changing balance

The Health and medical model wins as the ebb and flow have become more ebb and less flow.

The Incentive model wins as we are trapped by the other models.

The political model wins because we cannot hold to account anyone who decides and cannot determine who decides who decides because they make us think we did. 

Every one of the eight bn people on earth interprets each model based on bias, framed by their context, experience, and beliefs. 

So what happens when the model wins? Surely something else must come second, lose or be compromised.  Right now, as our models win the Earth is not compromised, as the Earth will remain here for another few billion years irrespective of what we do - it will just have a different balance. The compromise is the burden of our future generations from whom we have removed choice.  We have not questioned and determined what lifestyle they want; we have just focussed on the one we can get. 

Can the Earth also win if the model wins? Definitely, however, can further generations enjoy and have a life we expect if our models remain - that is a hope that we need to question.

 


@_Nat Zone

【講演案内】アフターコロナのトラスト形成 (2021-05-21)

第25回サイバー犯罪に関する白浜シンポジウム テー… The post 【講演案内】アフターコロナのトラスト形成 (2021-05-21) first appeared on @_Nat Zone.

第25回サイバー犯罪に関する白浜シンポジウム テーマ今こそ考えるサイバー空間の「信頼」~クラウドセキュリティとゼロトラストネットワーク~ の中で、「アフターコロナのトラスト形成」(2021年5月21日13:55~14:35)と題して講演します。また、その後引き続き「トラストなき時代のセキュリティ」と題して、上原 哲太郎 先生の司会で河野 省二 氏(日本マイクロソフト株式会社)と丸山 満彦 氏(PwCコンサルティング合同会社)とパネルディスカッションを行います。
全体のプログラムは、https://sccs-jp.org/symposium25/lecture/#lecture03 からアクセスしていただけます。(パネルの時間が間違っていますが…。)

それでは、オンラインでお会いしましょう。

2日目 13:55~14:35

アフターコロナのトラスト形成

崎村 夏彦 氏(NATコンサルティング合同会社)

新型コロナ禍により、多くの人々は好む好まざるに関わらず、インターネット上に生活の軸足を移さざるをえなくなりました。
サイバー大陸(第八大陸)への強制移住です。その結果、これまでの慣習にしたがった「信頼」のあり方の課題が噴出してきました。

このセッションでは、こうした課題の内主なものを取り上げ紹介するとともに、どのように解くのが良いと考えられているかを諸外国や標準化の動きを交えながら考察します。

The post 【講演案内】アフターコロナのトラスト形成 (2021-05-21) first appeared on @_Nat Zone.

Wednesday, 19. May 2021

Doc Searls Weblog

Making useful photographs

What does it mean when perhaps hundreds of thousands of one’s photos appear in articles, essays and posts all over the Web? It means they’re useful. That’s why I posted the originals in the first place, and licensed them to require only attribution. Because of that, I can at least guess at how many have […]

What does it mean when perhaps hundreds of thousands of one’s photos appear in articles, essays and posts all over the Web?

It means they’re useful. That’s why I posted the originals in the first place, and licensed them to require only attribution. Because of that, I can at least guess at how many have been put to use.

For one example subject, take Lithium, a metal in the periodic table. Lithium is making news these days, because it’s both scarce and required for the batteries of electric and hybrid vehicles. At issue especially is how and where lithium is extracted from the Earth. As Ivan Penn and Eric Lipton put it in The Lithium Gold Rush: Inside the Race to Power Electric Vehicles (6 May in The New York Times), extraction “might not be very green.”

But it is blue. Or turquoise. Or aqua. Or whatever colors you see in the photo above.

I took that shot on a 2010 flight over Nevada. Looking out the window, it’s hard to miss lakes of bright colors on the desert floor, looking like stained glass windows into the Earth. I didn’t know at the time that the puddles were lithium, but I did know they’d be useful when I published them, along with whatever information a little bit of research would reveal about them. After I did the research, I put 17 photos in an album on Flickr titled Lithium Mines in Nevada and added the same set to another album called Mines and Mining, which is now 329 photos long.

Also on that flight, which produced 130 photos now in an album called 2010_08_06 rno-phx-bos, other topics of interest are the Shoshone Mountains, Yucca Lake and Yucca Flat (with “subsidence craters” over underground nuclear bomb explosions), the Nevada Test Site, (where hundreds of atomic bomb tests took place, among other interesting things, “Doom Town” on Frenchman Lake, Broom Lake in Area 51, Creech Air Force Base (from which military drones are remotely controlled), Grand Canyon, and Buffalo at night. None of the photos of mine at those links (all in Wikipedia) are especially artistic. In fact most of them make me cringe today, because I hadn’t yet mastered Photoshop when I posted them in the first place. Back then I shot only .jpgs, rather than RAW photos, which means I can’t go back do much to improve them. But all are useful, especially to writers and publications covering the topic of lithium mining. For example, my photos of those lithium lakes appear in—

Biden clean energy talk fuels mining reform bills in E&E What an ancient lake in Nevada reveals about the future of tech in Fast Company TRANSITION TO ELECTRIC CARS NEED NOT DEMAND A TOXIC LITHIUM LEGACY, in Energy Mix Leading the Charge…To Lithium And Beyond? in Nevada Forward Lithium: Commodity Overview in Geology for Investors Lithium mining innovators secure investment from Bill Gates-led fund in Mining Technology The Path to Lithium Batteries: Friend or Foe? in Treehugger

And those are just the first six among 23,200 results in a search for my name + lithium. And those results are just from pubs that have bothered to obey my Creative Commons license, which only requires attribution. Countless others don’t.

Google also finds 57,400 results for my name + mining. On top of those, there are also thousands of other results for potash, river, geology, mining, mountains, dunes, desert, beach, ocean, hebrides, glacier, and other landforms sometimes best viewed from above. And that’s on top of more than 1500 photos of mine parked in Wikimedia Commons, of which many (perhaps most) are already in Wikipedia (sometimes in multiple places) or on their way there.

And those are just a few of the many subjects I’ve shot, posted and annotated to make them useful to the world. Which is why I’m guessing the number of photos actually being used is in the hundreds of thousands by now.

I have placed none of those photos in any of those places. I just put them up where they can easily be found and put to use. For example, when I shot Thedford, Nebraska, I knew somebody would find the photo and put it in Wikipedia.

Shots like these are a small percentage of all the photos I’ve taken over many decades. In fact, most of my photography is of people and scenes, not stuff like you find in the links above.

But apparently my main calling as a photographer is to push useful photos to the edge of the public domain, and to describe and tag them in ways that make them easy for researchers and journalists to find and use. And so far that has been a very successful strategy.

Addendum:::

So I have a camera question for the fellow photographers out there.

My main camera is a 2012-vintage Canon 5D Mark III , which replaced a 2005-vintage Canon 5D (source of the lithium lake shots), which replaced a Canon 30D of the same generation, and a Nikon Coolpix before that. All of these are retired or beat up now. Being um, resource constrained, every camera and lens I’ve used in this millennium I’ve either rented or bought used.

Now, out of great kindness, an old friend is giving me a Sony a7R that has been idle since she replaced it with a Sony a7Riii. I’ve played with her newer Sony, and really like how much lighter mirrorless full-frames can be. (And the a7R is lighter than the a7Riii.) The question now is what kind of lens I want to start with here, given that my budget is $0 (though I will spend more than that). The Sony equivalent of the lens I use most, a Canon 24-105 f4 L, runs >$1000, even used.

I suppose I could get non-Sony lenses for less, but … I’m not sure that’s the way to go. I’m kinda tempted to get a telephoto zoom or prime for the Sony and keep using the Canon for everything else. But then I’m carrying two cameras everywhere.

But I just looked at Ken Rockwell’s take on the Sony 24-105mm f/4
FE G OSS Full-Frame E-Mount
, which appears to outperform the Canon equivalent (two links back) so maybe I’ll find a bullet to bite, and spend the grand.

[25 May…] And I did that. The lens just arrived. Now I just need to match it up with a7R, which will probably happen next Tuesday. I trust you’ll see some results soon after that.


@_Nat Zone

国際金融協会(IIF)のAPACサミットでバンコク銀行頭取他とパネルディスカッショします(5/20, 5/21)

国際金融協会(International Inst… The post 国際金融協会(IIF)のAPACサミットでバンコク銀行頭取他とパネルディスカッショします(5/20, 5/21) first appeared on @_Nat Zone.

国際金融協会(International Institute of Finance, IIF)のアジア・パシフィックサミットに出演します。1日目の最後のセッション、5月20日11:20(日本時間)〜12:00 で、バンコク銀行頭取など錚々たる方々の間でのパネルディスカッションです。(英語です)

「Eコマースの未来」

新型コロナ は、オンライン商取引、新しい支払方法の採用、消費者の嗜好の大幅な変化などの傾向を劇的に加速させました。このことは、これらの変化がどの程度続くのか、あるいはさらに進化し続けるのか、企業が新技術を市場に投入する能力、そして中小企業がEコマースビジネスに移行したり再構築したりする際に必要なツールや技術について、重要な問題を提起しています。このセッションでは、デジタル化されたサービスをエンドユーザーや消費者に届けるための「ラストワンマイル」の課題についても考えます。
このセッションでは、デジタル化されたサービスをエンドユーザーや消費者に届ける「ラストマイル」の課題や、実店舗の未来、パンデミック後の世界でEコマースと従来の小売業がどのように融合するかについても考えます。

Brad Carr, Managing Director, Digital Finance, IIF (Moderator) Nat Sakimura, Chairman, OpenID Foundation Chartsiri Sophonpanich, President, Bangkok Bank Public Company Limited Kati Suominen, Founder and Chief Executive Officer, Nextrade Group

このサミットは、このパネル以外にも見どころが沢山です。たとえば、2日目の朝一番には金融庁の氷見野長官が登場します。

お申し込みはこちらのページの一番下から→申し込みページ

プログラム→ プログラム (PDF)

The post 国際金融協会(IIF)のAPACサミットでバンコク銀行頭取他とパネルディスカッショします(5/20, 5/21) first appeared on @_Nat Zone.

Phil Windley's Technometria

Can the Digital Future Be Our Home?

Summary: This post features three fantastic books from three great, but quite different, authors on the subject of Big Tech, surveillance capitalism, and what's to be done about it. I recently read Shoshana Zuboff's book on surveillance capitalism. Not only is the book thought provoking, but Zuboff's writing verges on the poetic at times, making it a delightful read. In her opening c

Summary: This post features three fantastic books from three great, but quite different, authors on the subject of Big Tech, surveillance capitalism, and what's to be done about it.

I recently read Shoshana Zuboff's book on surveillance capitalism. Not only is the book thought provoking, but Zuboff's writing verges on the poetic at times, making it a delightful read. In her opening chapter she asks the question "Can the digital future be our home?"

This question is perhaps one of the most important of our age. More and more of our lives are being intermediated by digital systems. And yet those systems are not ours, but rather belong to the companies that provide them. And our experience on them is predicated on the goals, desires, and needs of those companies, not ours. I call these systems "administrative" because they are built to administer our experience in a particular domain for the administrator's specific purposes.

The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power by Shoshana Zuboff

The challenges to humanity posed by the digital future, the first detailed examination of the unprecedented form of power called "surveillance capitalism," and the quest by powerful corporations to predict and control our behavior. In this masterwork of original thinking and research, Shoshana Zuboff provides startling insights into the phenomenon that she has named surveillance capitalism. The stakes could not be higher: a global architecture of behavior modification threatens human nature in the twenty-first century just as industrial capitalism disfigured the natural world in the twentieth.

Zuboff makes a number of compelling arguments about why surveillance capitalism represents a significant threat to humanity's future. An overarching conclusion is that by putting everyone inside their administrative systems to make our lives legible to their surveillance, these companies become tyrants.

[T]yranny is the obliteration of politics. It is founded on its own strain of radical indifference in which every person, except the tyrant, is understood as an organism among organisms in an equivalency of Other-Ones.

Contrary to what many might believe, the obliteration of politics is not a good thing. As we discovered a few issues ago (see Legitimacy and Decentralized Systems), politics is how decentralized, democratic systems achieve legitimacy and coherence. Getting rid of politics requires putting everyone and everything in the centralized administrative system of the surveillance capitalist—making them subject to the dictates of the tyrant who has radical indifference to their autonomy, individuality, and humanity.

Zuboff's statement echos David Graeber's discussion of bureaucracy in The Utopia of Rules. Bureaucratic interactions are simple and predictable. But they are soulless. They are transactional and cannot provide the basis for authentic digital relationships (see Authentic Digital Relationships ).

The Utopia of Rules: On Technology, Stupidity, and the Secret Joys of Bureaucracy by David Graeber

Where does the desire for endless rules, regulations, and bureaucracy come from? How did we come to spend so much of our time filling out forms? And is it really a cipher for state violence? To answer these questions, the anthropologist David Graeber—one of our most important and provocative thinkers—traces the peculiar and unexpected ways we relate to bureaucracy today, and reveals how it shapes our lives in ways we may not even notice...though he also suggests that there may be something perversely appealing—even romantic—about bureaucracy.

Living our lives inside the administrative systems of Big Tech is akin to living your life inside an amusement park. Not altogether unpleasant, but a far cry from authentic. Stippled with moments of joy, but devoid of real happiness and freedom. Treated identically and transactionally despite pretensions to personalization.

In How to Destroy Surveillance Capitalism, Cory Doctorow argues that while Zuboff's observations are not incorrect, her conclusions about what constitutes surveillance capitalism's real dangers are mistaken. Where Zuboff sees companies who are getting better and better at predicting and controlling our actions, Doctorow sees companies selling the power to persuade, poorly. The real harm is the surveillance, not mind control.

How to Destroy Surveillance Capitalism by Cory Doctorow

For years, we've been hearing about the ills of surveillance capitalism --- the business of extracting, collecting, and selling vast reams of user data that exploded with the rise of tech giants like Google, Facebook, and Amazon. But what if everything we've been hearing is wrong? What if surveillance capitalism is not some rogue capitalism or a wrong turn taken by some misguided corporations? What if the system is working exactly as intended—and the only hope of restoring an open web is to take the fight directly to the system itself?

Zuboff's conclusion that surveillance capitalism is a new "rogue" form of capitalism leaves us with little recourse but to regulate the ills that surveillance capitalists bring about. Not unreasonably, Zuboff's prescription for this predicament is to protect, trust, and utilize democratic processes—to collectively push back. To not let our cynicism dissuade us or cause us to lose hope.

But, merely regulating a big monopoly only further entrenches it, locking the world into the status quo. If we want to destroy surveillance capitalism, Cory argues, we have to break it up and decentralize, making "big tech small again." Ultimately, the choice is to fix Big Tech or fix the internet. Cory argues for the second and I'm on board.

Fixing the internet is hard, but not impossible. Cory references Lawrence Lessig, saying "our lives are regulated by four forces: law (what's legal), code (what's technologically possible), norms (what's socially acceptable), and markets (what's profitable)." We can bring all four to bear on this problem.

Antitrust, post Reagan, has lost its teeth and come to focus only on consumer harm instead of other anti-competitive behaviors like buying up large rivals and new competitors. If the problem with "Big Tech" is that it is "big" then restructuring antitrust laws to break up large tech companies is a critical tool.

Many will fear that breaking up big tech will diminish the fruits of the digital world we've come to enjoy, and even rely on. Centralization, they will say, is the only way to safely and efficiently build messaging platforms, app stores, social networks, and other features of Web 2.0 that we've come to enjoy.

This is where Lessig's other three forces come into play. As I've written, in numerous ways, the means exist to decentralize most of the centralized Web 2.0 platforms (i.e. it's "technologically possible" in Lessig's words). The internet itself and more recent decentralized networks like Bitcoin and Ethereum show that large, decentralized systems can achieve legitimacy to accomplish global goals.

Beyond tech, I have hope that norms are changing. People are more aware and wary of the dangers of surveillance and the need for better online privacy. Collecting data is becoming less socially acceptable. Security breeches affect more and more people, waking them up to the problem of companies collecting and holding large caches of personal data. And competitors to big tech with decentralized solutions are always emerging. A little antitrust help could be what it takes to make them viable.

There's no single act that's going to change the way things work now. Getting Congress to act on antitrust requires a big shift in norms. Changing norms requires new technical possibilities, new applications, and, frankly, more privacy problems. Change is predicated on a web of interrelated actions that we must iterate over.

Returning to Zuboff's opening question: "Can the digital future be our home?" Fixing Big Tech just leaves us where we're at, with slightly fewer problems. It's a dead end road that doesn't lead to a digital home. But fixing the internet, redecentralizing it, promises a future where we can live authentic digital lives that compliment our physical lives. I choose to fight for that future.

Tags: identity surveillance+capitalism decentralization

Monday, 17. May 2021

Damien Bod

Securing OAuth Bearer tokens from multiple Identity Providers in an ASP.NET Core API

This article shows how to secure and use different APIs in an ASP.NET Core API which support OAuth access tokens from multiple identity providers. Access tokens from Azure AD and from Auth0 can be be used to access data from the service. Each API only supports a specific token from the specific identity provider. Microsoft.Identity.Web […]

This article shows how to secure and use different APIs in an ASP.NET Core API which support OAuth access tokens from multiple identity providers. Access tokens from Azure AD and from Auth0 can be be used to access data from the service. Each API only supports a specific token from the specific identity provider. Microsoft.Identity.Web is used to implement the access token authorization for the Azure AD tokens and the default authorization is used to support the Auth0 access tokens.

Code: https://github.com/damienbod/SeparatingApisPerSecurityLevel

Blogs in this series

Securing multiple Auth0 APIs in ASP.NET Core using OAuth Bearer tokens Securing OAuth Bearer tokens from multiple Identity Providers in an ASP.NET Core API

Setup

An API ASP.NET Core application is created to implement the multiple APIs and accept access tokens created by Auth0 and Azure AD. The access tokens need to be validated and should only work for the intended purpose for which the access token was created. The Azure AD API is used by an ASP.NET Core Razor page application which requests an user access token with the correct scope to access the API. Two Azure AD App registrations are used to define the Azure AD setup. The Auth0 application is implemented using a Blazor server hosted application and accesses the two Auth0 APIs, See the pervious post for details.

To support the multiple identity providers, multiple schemes are used. The Auth0 APIs use the default scheme definition for JWT Bearer tokens and the Azure AD uses a custom named scheme. It does not matter which scheme is used for which as long as the correct scheme is defined on the controller securing the API. The AddMicrosoftIdentityWebApiAuthentication method takes the scheme and the configuration name as a optional parameter. The Azure AD configuration is defined like any standard Azure AD API in ASP.NET Core.

public void ConfigureServices(IServiceCollection services) { // Adds Microsoft Identity platform (AAD v2.0) // support to protect this Api services.AddMicrosoftIdentityWebApiAuthentication( Configuration, "AzureAd", "myADscheme"); // Auth0 API configuration=> default scheme services.AddAuthentication(options => { options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme; options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme; }).AddJwtBearer(options => { options.Authority = "https://dev-damienbod.eu.auth0.com/"; options.Audience = "https://auth0-api1"; }); services.AddSingleton<IAuthorizationHandler, UserApiScopeHandler>(); // authorization definitions for the multiple Auth0 tokens services.AddAuthorization(policies => { policies.AddPolicy("p-user-api-auth0", p => { p.Requirements.Add(new UserApiScopeHandlerRequirement()); // Validate id of application for which the token was created p.RequireClaim("azp", "AScjLo16UadTQRIt2Zm1xLHVaEaE1feA"); }); policies.AddPolicy("p-service-api-auth0", p => { // Validate id of application for which the token was created p.RequireClaim("azp", "naWWz6gdxtbQ68Hd2oAehABmmGM9m1zJ"); p.RequireClaim("gty", "client-credentials"); }); }); services.AddControllers(options => { var policy = new AuthorizationPolicyBuilder() .RequireAuthenticatedUser() .Build(); options.Filters.Add(new AuthorizeFilter(policy)); }); }

The Configure method uses the UseAuthentication method to add the middleware for the APIs.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { // ... app.UseRouting(); app.UseAuthentication(); app.UseAuthorization(); app.UseEndpoints(endpoints => { endpoints.MapControllers(); }); }

The AzureADUserOneController class is used to implement the API for the Azure AD access tokens. The AuthorizeForScopes attribute from Microsoft.Identity.Web is used to validate the Azure AD App registration access token and define the scheme required for the validation. The scope name must match the Azure App registration definition.

using System.Collections.Generic; using Microsoft.AspNetCore.Authorization; using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.Logging; using Microsoft.Identity.Web; namespace MyApi.Controllers { [AuthorizeForScopes(Scopes = new string[] { "api://72286b8d-5010-4632-9cea-e69e565a5517/user_impersonation" }, AuthenticationScheme = "myADscheme")] [ApiController] [Route("api/[controller]")] public class AzureADUserOneController : ControllerBase { private readonly ILogger<UserOneController> _logger; public AzureADUserOneController(ILogger<UserOneController> logger) { _logger = logger; } [HttpGet] public IEnumerable<string> Get() { return new List<string> { "AzureADUser one data" }; } } }

The UserOneController implements the Auth0 user access token API. Since the default scheme is used, no scheme definition is required. The authorization policy is used to secure the API which validates the scope and the claims for this API.

using System.Collections.Generic; using Microsoft.AspNetCore.Authorization; using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.Logging; namespace MyApi.Controllers { [Authorize(Policy = "p-user-api-auth0")] [ApiController] [Route("api/[controller]")] public class UserOneController : ControllerBase { private readonly ILogger<UserOneController> _logger; public UserOneController(ILogger<UserOneController> logger) { _logger = logger; } [HttpGet] public IEnumerable<string> Get() { return new List<string> { "user one data" }; } } }

When the API application is started the APIs can be used and a swagger UI implemented using Swashbuckle was created to display the different APIs. Each API will only work with the correct access token. The different UIs can use the APIs and data is returned.

Links

https://auth0.com/docs/quickstart/webapp/aspnet-core

https://docs.microsoft.com/en-us/aspnet/core/security/authorization/introduction

Open ID Connect

Securing Blazor Web assembly using Cookies and Auth0

Sunday, 16. May 2021

Hyperonomy Digital Identity Lab

NETAGO Downtime – 2021-05-15 & 16

Hardwired ThinkPad laptop: 81.01% downtime (same laptop used for prior NUM reports) WiFi Wireless Lenovo laptop: 3.78% downtime
Hardwired ThinkPad laptop: 81.01% downtime (same laptop used for prior NUM reports) WiFi Wireless Lenovo laptop: 3.78% downtime netuptime-20210516-075453-tp-hardwiredDownload netuptime-20210514-061223-lenovo-wifiDownload

Friday, 14. May 2021

Hyperonomy Digital Identity Lab

NETAGO Downtime – 2021-05-14 AM

Hardwired ThinkPad laptop: 77.96% downtime (same laptop used for prior NUM reports) WiFi Wireless Lenovo laptop: 2.99% downtime
Hardwired ThinkPad laptop: 77.96% downtime (same laptop used for prior NUM reports) WiFi Wireless Lenovo laptop: 2.99% downtime netuptime-20210513-230229-tp-hardwiredDownload netuptime-20210513-230235-lenovo-wifiDownload

Thursday, 13. May 2021

Here's Tom with the Weather

Betty's Funeral Service

My mother Betty passed away Friday night and her service is Saturday at 2pm at the Kingwood Funeral Home. This is one of my favorite pictures of her with her mother and my brother David in downtown San Francisco. Growing up, I always remember that there was a set of golf clubs in the garage but it seemed like they were never used. Luckily, this week, I listened to a Christmas audio message

My mother Betty passed away Friday night and her service is Saturday at 2pm at the Kingwood Funeral Home. This is one of my favorite pictures of her with her mother and my brother David in downtown San Francisco.

Growing up, I always remember that there was a set of golf clubs in the garage but it seemed like they were never used. Luckily, this week, I listened to a Christmas audio message from 1971 that my parents had sent to my grandparents. My mom said that the day before, Baxter had taken her to the golf course at Heather Farm and they golfed nine holes. She said on the first hole, she amused several bystanders trying to get out of the sand trap and they decided to quit keeping score from there. She said she learned golf was not her sport and would find something else. She did find tennis. I was glad to play that sport with her and the family.


Hyperonomy Digital Identity Lab

NETAGO Downtime – 2021-05-13 AM – 88.81%

Over the last approximately 7.5 hours, the NETAGO internet service in Bindloss, Alberta was down 88.81% of the time according to the Net Uptime Monitor (NUM) app. This is not acceptable. Net Uptime Monitor Failure Log (NetUptimeMonitor.com)Licensed to Michael Herman … Continue reading →

Over the last approximately 7.5 hours, the NETAGO internet service in Bindloss, Alberta was down 88.81% of the time according to the Net Uptime Monitor (NUM) app. This is not acceptable.

Net Uptime Monitor Failure Log (NetUptimeMonitor.com)
Licensed to Michael Herman

=======================================

2021-05-12 10:19:45 PM Log Start

Failure Start Length
2021-05-12 10:20:33 PM 0:00:05
2021-05-12 10:20:52 PM 0:03:23
2021-05-12 10:24:47 PM 0:01:32
2021-05-12 10:26:54 PM 0:04:16
2021-05-12 10:31:32 PM 0:07:08
2021-05-12 10:38:48 PM 0:01:30
2021-05-12 10:40:26 PM 0:10:28
2021-05-12 10:51:22 PM 0:03:33
2021-05-12 10:55:03 PM 0:06:38
2021-05-12 11:01:50 PM 0:13:03
2021-05-12 11:16:13 PM 0:09:05
2021-05-12 11:26:56 PM 0:10:07
2021-05-12 11:37:12 PM 0:04:32
2021-05-12 11:44:15 PM 0:12:41
2021-05-12 11:58:03 PM 0:03:08
2021-05-13 12:05:52 AM 0:04:00
2021-05-13 12:10:00 AM 0:09:12
2021-05-13 12:19:20 AM 0:03:58
2021-05-13 12:23:26 AM 0:06:59
2021-05-13 12:32:56 AM 0:13:24
2021-05-13 12:47:53 AM 0:12:56
2021-05-13 1:00:57 AM 0:01:34
2021-05-13 1:03:24 AM 0:01:54
2021-05-13 1:07:04 AM 0:21:49
2021-05-13 1:29:07 AM 0:00:23
2021-05-13 1:29:39 AM 0:02:10
2021-05-13 1:31:57 AM 0:01:56
2021-05-13 1:34:01 AM 0:01:06
2021-05-13 1:35:15 AM 0:00:30
2021-05-13 1:36:19 AM 0:05:39
2021-05-13 1:42:07 AM 0:00:36
2021-05-13 1:42:51 AM 0:12:26
2021-05-13 1:56:36 AM 0:04:57
2021-05-13 2:01:41 AM 0:11:08
2021-05-13 2:12:58 AM 0:04:35
2021-05-13 2:20:10 AM 0:21:12
2021-05-13 2:41:44 AM 0:06:45
2021-05-13 2:48:44 AM 0:06:04
2021-05-13 2:57:25 AM 0:05:49
2021-05-13 3:04:02 AM 0:00:33
2021-05-13 3:04:56 AM 0:00:05
2021-05-13 3:05:23 AM 0:04:44
2021-05-13 3:10:22 AM 0:07:22
2021-05-13 3:17:52 AM 0:07:44
2021-05-13 3:27:15 AM 0:18:57
2021-05-13 3:46:20 AM 0:09:16
2021-05-13 3:58:39 AM 0:00:51
2021-05-13 4:01:55 AM 0:06:26
2021-05-13 4:08:42 AM 0:19:29
2021-05-13 4:28:20 AM 0:10:05
2021-05-13 4:40:36 AM 0:11:36
2021-05-13 4:52:21 AM 0:02:01
2021-05-13 4:54:36 AM 0:03:54
2021-05-13 4:58:39 AM 0:03:10
2021-05-13 5:01:57 AM 0:07:02
2021-05-13 5:09:33 AM 0:07:25
2021-05-13 5:17:19 AM 0:02:02
2021-05-13 5:19:36 AM 0:00:33
2021-05-13 5:20:17 AM 0:01:09
2021-05-13 5:23:24 AM 0:01:32
2021-05-13 5:25:05 AM 0:03:57
2021-05-13 5:29:23 AM 0:01:28
2021-05-13 5:30:59 AM 0:10:35
2021-05-13 5:42:40 AM 0:01:57
2021-05-13 5:46:10 AM 0:00:51
2021-05-13 5:47:09 AM 0:00:13

2021-05-13 5:48:22 AM 0:08:06

2021-05-13 5:56:36 AM Log End


Monitor Duration 7:36:50
Failure Summary:
Count 67
Total Downtime 6:45:44
% Downtime 88.81
Minimum Length 0:00:05
Maximum Length 0:21:49
Average Length 0:06:03

Wednesday, 12. May 2021

MyDigitalFootprint

who wins when our diversity creates less diversity?

The Media wins by playing with us When Education wins, everyone wins I can win but the self interest distroys more We win by being one together in our diversity It is not we lose by doing nothing, someone else gains more Our paradox is that the more diversity we have, the less diverse we become.   
The Media wins by playing with us
When Education wins, everyone wins
I can win but the self interest distroys more
We win by being one together in our diversity
It is not we lose by doing nothing, someone else gains more Our paradox is that the more diversity we have, the less diverse we become.   

Hyperonomy Digital Identity Lab

NETAGO Downtime – 2021-05-12 Early Morning – 93.6%

Net Uptime Monitor Failure Log (NetUptimeMonitor.com)Licensed to Michael Herman ======================================= 2021-05-12 3:24:40 AM Log Start Failure Start Length2021-05-12 3:26:48 AM 0:00:052021-05-12 3:27:08 AM 0:00:162021-05-12 3:28:06 AM 0:06:222021-05-12 3:34:36 AM 0:01:042021-05-12 3:35:48 AM 0:05:232021-05-12 3:43:30 AM 0:27:342021-05-12 4:11:13 AM 0:00:292021-05-12 4:11:56 … Conti

Net Uptime Monitor Failure Log (NetUptimeMonitor.com)
Licensed to Michael Herman

=======================================

2021-05-12 3:24:40 AM Log Start

Failure Start Length
2021-05-12 3:26:48 AM 0:00:05
2021-05-12 3:27:08 AM 0:00:16
2021-05-12 3:28:06 AM 0:06:22
2021-05-12 3:34:36 AM 0:01:04
2021-05-12 3:35:48 AM 0:05:23
2021-05-12 3:43:30 AM 0:27:34
2021-05-12 4:11:13 AM 0:00:29
2021-05-12 4:11:56 AM 0:02:18
2021-05-12 4:14:55 AM 0:02:05
2021-05-12 4:17:09 AM 0:05:56
2021-05-12 4:23:13 AM 0:56:42
2021-05-12 5:20:23 AM 0:00:23
2021-05-12 5:21:46 AM 0:00:33
2021-05-12 5:22:27 AM 0:07:16
2021-05-12 5:29:51 AM 0:04:38
2021-05-12 5:34:38 AM 0:17:12
2021-05-12 5:51:58 AM 0:07:41
2021-05-12 5:59:54 AM 0:00:15
2021-05-12 6:02:15 AM 0:00:27
2021-05-12 6:02:50 AM 0:04:07
2021-05-12 6:07:37 AM 0:21:43

2021-05-12 6:29:28 AM 0:07:42

2021-05-12 6:37:22 AM Log End


Monitor Duration 3:12:41
Failure Summary:
Count 22
Total Downtime 3:00:21
% Downtime 93.60
Minimum Length 0:00:05
Maximum Length 0:56:42
Average Length 0:08:11


Net Uptime Monitor (NUM)

The Net Uptime Monitor What it does… Is your internet connection unreliable? You’ve probably called your internet provider’s support line and maybe they were able to help you, maybe they even sent out a tech to look at it. But … Continue reading →

The Net Uptime Monitor


What it does…


Is your internet connection unreliable? You’ve probably called your internet provider’s support line and maybe they were able to help you, maybe they even sent out a tech to look at it. But all too often the response is “Well, it’s working fine now!”


The Net Uptime Monitor alerts you to failures in your internet connection and documents the exact time and length of those failures. This failure log will help your provider troubleshoot the problem – after it helps you convince them it’s not your imagination! Net Uptime Monitor is designed to be as simple as possible and accomplish this one purpose accurately and thoroughly with the least effort from you.


How it works…


Net Uptime Monitor (NUM) uses the “Ping” command to test the response from three public servers operated by Google, Level 3, and OpenDNS. (See “What’s a Ping?” below for an explanation.) Each server is pinged in turn at an interval that you can set – normally five seconds. By default, NUM waits 200 milliseconds (2/10 of a second) for the server to respond – at least 3 times as long as a typical broadband internet connection should take.


NUM pings one server at a time; if the server responds, NUM waits the test interval, then pings the next server. If the server doesn’t respond, NUM immediately tries the next server, then the next. If any of the servers respond, then your connection must be working. Only when all three servers fail to respond does NUM determine that your connection is down.


By using three servers, NUM ensures that the problem isn’t just with the server or with some connection on the way to that server, or that the server isn’t momentarily slow or congested.


NUM can detect failures as short as a couple of seconds in length, but you can decide how long a failure must be before it really counts. A very short failure of a second or so is not likely to affect your use of the net and isn’t of any real concern. You can set how long a failure must be before NUM alerts you to it and records the failure in its failure log.


Connection is up, no previous failures…

Connection is down, one previous failure…

The display shows the names and IP addresses of each server. The indicator “light” flashes yellow when the ping is sent and shows green for a successful response. The response time of the last ping is shown. When the response time exceeds the time set for “Wait for Ping Response”, the indicator turns red to show no response from that server.


If your connection fails, the current fail length is displayed in red. When the length of the failure exceeds your setting for “Log Failure If Longer Than”, NUM plays an alert sound and writes the failure information into its log.


The display also shows the monitored time (how long the monitor has been running), the time since the last logged failure (up time), the start time and length of the last logged failure, and the total count of logged failures since NUM was started. The current settings for the test interval and the minimum failure length to be logged are shown at the bottom of the display.


Click the minimize button on the NUM window to hide the display. NUM disappears into your system tray in the “notifications area”. The NUM icon is shown in the notification – you can hover over the icon to see the current time since the last failure (“Up Time”) or click the icon to restore the display. In the Settings, you can choose to have a “failure alert” sound play, and/or have the NUM window “pop up”, if a connection failure longer than your minimum setting occurs.


The Log


NUM keeps a log of results in a text file. You can view the current log at any time by clicking the “View Log” button. The log is displayed in a separate window. NUM will continue to update the log even while you are viewing it.
Because the log is a plain text file, you can open it outside of the NUM program. It will open in Notepad or your default text editor, so you can easily edit or print the log.


The log records the start and end time of the monitoring and each failure start time and length. A summary shows the total monitoring time, failure count, total down time, percentage of down time, and the minimum, maximum, and average failure lengths. Here’s an example:

Net Uptime Monitor Failure Log (NetUptimeMonitor.com)
Licensed to Example User

=======================================

8/17/2015 8:44:28 AM Log Start

Failure Start Length
8/17/2015 1:44:25 PM 0:00:44
8/17/2015 1:49:53 PM 0:00:36

8/17/2015 1:52:39 PM 0:01:59

8/18/2015 12:13:17 AM Log End
Monitor Duration 15:28:46
Failure Summary:
Count 3
Total Downtime 0:03:20
% Downtime 0.36
Minimum Length 0:00:36
Maximum Length 0:01:59

Average Length 0:01:06

The example shows date and time in US English format; your log will use the format for your region.
The log files are saved in a folder of your choice; the default is your Documents folder. You can choose a different folder in the Settings.


Also in the Settings, there are two options for the log file:
1) New File Each Run
A new file is created each time NUM starts. Each log file is named with the date and time NUM was started so that they will appear in your directory in chronological order. The file name is in the form of “NetUptime 20110805 134243.txt”. In this example, the date is August 10, 2015 – 20150810 – and the time is 1:42:43 PM – 134243.
2) Add to Existing File
Each new log is added to the same single file. The file name is always NetUptime.txt. As long as that file exists in the folder where you have chosen to save the log file, NUM will keep adding to it. If the file doesn’t exist, i.e. it’s been deleted, moved, or renamed, NUM will start a new file.


The Settings

Click the “Change Settings” button on the NUM display to open the Settings window. There are several settings available:


Startup Settings…


· Start when Windows Starts? – Check the box and NUM will automatically start when your computer starts. Uncheck the box and you can start NUM when you want by clicking its desktop icon. The default on installation is checked – NUM starts automatically.
· Start Minimized in Tray? – Check the box and NUM will be minimized in the system tray automatically when it starts. The default on installation is unchecked – NUM starts with the main form displayed.
Test Settings…
· Test Interval – how many seconds between ping tests when the servers are responding. Five seconds is the default. It is possible that NUM will miss a failure that is shorter than the time between tests, so if your connection has very frequent failures of just a few seconds you might choose a shorter test interval. If you don’t have many failures, you may want to test less often. Most connection problems result in less frequent but longer failures, so five seconds is a good choice for most users.
· Wait for Ping Response – the length of time NUM waits for a response after sending a ping. The default setting of 200 milliseconds is recommended for normal situations. If you have a slower internet connection, such as a dialup or mobile connection, or are in a remote area where response is typically slow, you can set the wait time for up to 1500 milliseconds (1.5 seconds). To help you find the best setting for your situation, set the wait time to 1500 milliseconds and observe the ping response times NUM displays when your connection is working normally. Set the wait time to about 1.5 times the typical ping response times you see for efficient detection of outages.
· Change Target Servers – click to open the Target Servers window.

You can edit the IP Address and Name of any of the three servers. Click the Test button to try that server, verifying that it responds and checking the response time.


The default target servers (Google, Level 3, OpenDNS) were selected for their performance and very high reliability. You should only use a different server if you find that one of these servers does not respond reliably in your particular situation. Click “Restore Defaults” to reset the Target Servers to their original values. Changes to the Target Servers take effect the next time the program starts.


Alert and Log Settings…


· Pop Up on Failure? – Check the box and the NUM form will pop up from the system tray when there is a failure. Uncheck the box and NUM will continue to log and alert but it will stay minimized during a failure. The default on installation is checked – if NUM is minimized to the system tray, the main NUM form will be displayed when a failure is logged.
· Alert and Log Failure If Longer Than – the minimum failure length that will be counted, both for the log and the alert of a failure. Five seconds is the default setting.
· Log File Location – the folder where the logs will be stored. Click the “Select Folder” button to browse to the folder you want. The log for the current run of NUM is already started, so a change in this setting will not take effect until the next time you run NUM.
· Log File Option – New File Each Run (the default) or Add to Existing File. See previous section “The Log” for a more detailed explanation.
· Choose Failure Alert Sound – choose the sound NUM makes when a failure is counted. The sound plays when you choose its button so you can preview each one. Choose “None” to silence the alert. Choose “Custom” and click the Select File button to use any .WAV sound file on your system. The default on installation is the “Short” sound.
· Play Reconnect Sound – NUM can play a sound when your internet reconnects after a failure. Choose “None” to silence the reconnect sound. Choose “Custom” and click the Select File button to use any .WAV sound file on your system.


Combine Settings for “Invisible” Operation


NUM can do its job without showing itself or alerting the user to its operation in any way. Choose these settings:
· Start when Windows Starts? – checked.
· Start Minimized in Tray? – checked.
· Pop Up On Failure – unchecked.
· Choose Failure Alert Sound – None.
· Choose Reconnect Sound – None.
With this combination of settings, the user need never be aware of NUM. This is useful in a support situation where you are installing NUM on a computer you aren’t personally using.


What’s a Ping?


“Ping” is a command available on all kinds of computers that tests whether another computer on the network will respond to your computer. It’s named after the sound of submarine sonar systems – they send out a “ping” sound which bounces off their target and they listen for that echo, locating their target. The internet “ping” works in a similar way. You name your target, an internet server, and “ping” it. The ping command and response looks like this (in a DOS command window):


C:\ ping google.com

Pinging google.com [74.125.224.84] with 32 bytes of data:
Reply from 74.125.224.84: bytes=32 time=30ms TTL=54
Reply from 74.125.224.84: bytes=32 time=31ms TTL=54
Reply from 74.125.224.84: bytes=32 time=31ms TTL=54
Reply from 74.125.224.84: bytes=32 time=31ms TTL=54

Ping statistics for 74.125.224.84:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 30ms, Maximum = 31ms, Average = 30ms

A ping command actually generates four requests and the server replies four times. Each response is timed in thousandths of a second (ms = milliseconds). Here we see that the server at google.com responded in about 31/1000 or 3/100 of a second. The internet is fast! – when everything’s working.


Licensing


A license for Net Uptime Monitor removes the time limits from the trial version and lets you use the full program on one computer. To purchase a license or register your license, just click “Trial Version – Click to Register or Purchase License” at the bottom of the NUM main form. If you have your license, enter the License Key code you’ve received and click Register. If you need a license, click Purchase a License to visit our web site and make your purchase.
If you have already registered your copy of NUM, your name and email are shown on the main form. Click the License Info button to see your license key.


Moving to a New Computer or Installing a New Operating System


You must unregister your license before you replace your computer or install a new version of Windows. This will make your license key available again to use on your new system. Just click License Info, click Print This Form to make sure you’ll have the license key, then click Unregister License. The program will go back to Trial mode. You can then reuse your license key to register NUM on any computer.

Tuesday, 11. May 2021

MyDigitalFootprint

Dashboards - we love them, but why do they love us?

Subject: Agenda item for our away day on strategy and scenarios To: CEO and senior exec team We should congratulate ourselves on the progress made, however as your CDO, I am now going to make a case that we measure too much, have too much data and that as a team, we should reflect on the next thing that data can support us in! We have bought into “Data is the new oil,” and whilst we know th


Subject: Agenda item for our away day on strategy and scenarios

To: CEO and senior exec team

We should congratulate ourselves on the progress made, however as your CDO, I am now going to make a case that we measure too much, have too much data and that as a team, we should reflect on the next thing that data can support us in!

We have bought into “Data is the new oil,” and whilst we know the analogy breaks down below the veneer, the message is beautifully simple and has empowered the change to a data and digital business. The global pandemic has accelerated our adoption and transformation, and we are in a better place than March 2020. However, sticking with oil, we know that the extraction process has downsides, including carbon release, messy, and difficulty locating economic wells.   Amongst data’s most significant downsides are legal liabilities, noise and the wrong data. 

I can easily hide data’s downsides through dashboards.  Our dashboards are based on trickle-down KPI and objectives from our strategic plan.  We hand out SMART objectives, but such objectives fundamentally assume that we are and continue to do the right thing.  We gather data and post-analysis present it as trending graphs or red, amber, green dashboards, aiming for everything to be going in the right direction or green.  Green only means we have no negative variance between our plan and the actual. Organisationally we are incentivised to achieve green at any unknown cost or consequence.  Trending analysis can always be generated through the selection of data to fit the story. 

Right now, we are using data to measure, analyse and inform. We have better control of variance across our entire operations and ecosystem than at any previous point in our history. We increasingly have to depend on dashboards to manage the complicated and complex, remain informed, and determine where our energies should be focussed.  Without a doubt, this should remain and continue to be improved, but we are already on a diminishing return with the data we collect due to the increase in noise over signal and should aim to cull than add. 

As an agenda item; should we introduce the colour BLUE into our reporting, trends, and dashboards to reduce reporting and data?  The traditional traffic lights remain the same; blue is not replacing any of them; it can become the one that allows us to know we are doing the right thing and not just doing the wrong thing that is in the plan in the most efficient and effective way (can be green or red).

As a team, we have to feed our sense of enquiry, and our existing data gathering, analysis, and reporting do not value the complexity we are faced with. More data does not solve complexity. Data has allowed us to evolve from finance data as the most critical decision-making source to become more sensitive.  Whilst we have far more data, it is still narrow, and we should consider how we prepare for the next evolution, which will not be more of the same data.  Following on from customer data and the start in ESG, the next data set we are being mandated to report on is Human Capital.  

Human capital reporting opens up our ability to sense what our employees and ecosystem are sensing, seeing and implying; helping us to determine, using technologies such as sentiment analysis, if we are doing the right thing. Where are issues that are not on the dashboard are occurring? What, as yet, unidentified tensions and conflicts are created by our current trickle-down objective/ incentive system? However, as you can imagine, big brother, privacy, and trust are foundational issues we need to discuss front up before this next evolutionary step hits us.  This next phase of our data journey means we will find it harder to hide in the selection of data for trends and dashboards or just seek the right trend or green. This is more data, but different data and it will fill a gap in our knowledge,  meaning we will be better informed about complex decisions. 

I would like to present for 15 minutes on this topic and host a 45-minute debate with your approval.

Your CDO


Monday, 10. May 2021

Damien Bod

Present and Verify Verifiable Credentials in ASP.NET Core using Decentralized Identities and MATTR

This article shows how use verifiable credentials stored on a digital wallet to verify a digital identity and use in an application. For this to work, a trust needs to exist between the verifiable credential issuer and the application which requires the verifiable credentials to verify. A blockchain decentralized database is used and MATTR is […]

This article shows how use verifiable credentials stored on a digital wallet to verify a digital identity and use in an application. For this to work, a trust needs to exist between the verifiable credential issuer and the application which requires the verifiable credentials to verify. A blockchain decentralized database is used and MATTR is used as a access layer to this ledger and blockchain. The applications are implemented in ASP.NET Core.

The verifier application Bo Insurance is used to implement the verification process and to create a presentation template. The application sends a HTTP post request to create a presentation request using the DID Id from the OIDC credential Issuer, created in the previous article. This DID is created from the National Driving license application which issues verifiable credentials and so a trust needs to exist between the two applications. Once the credentials have been issued to a holder of the verifiable credentials and stored for example in a digital wallet, the issuer is no longer involved in the process. Verifying the credentials only requires the holder and the verifier and the decentralized database which holds the digital identities and documents. The verifier application gets the DID from the ledger and signs the verify request. The request can then be presented as a QR Code. The holder can scan this using a MATTR digital wallet and grant consent to share the credentials with the application. The digital wallet calls the callback API defined in the request presentation body and sends the data to the API. The verifier application hosting the API would need to verify the data and can update the application UI using SignalR to continue the business process with the verified credentials.

Code https://github.com/swiss-ssi-group/MattrGlobalAspNetCore

Blogs in the series

Getting started with Self Sovereign Identity SSI Create an OIDC credential Issuer with MATTR and ASP.NET Core Present and Verify Verifiable Credentials in ASP.NET Core using Decentralized Identities and MATTR Verify vaccination data using Zero Knowledge Proofs with ASP.NET Core and MATTR

Create the presentation template for the Verifiable Credential

A presentation template is required to verify the issued verifiable credentials stored on a digital wallet.

The digital identity (DID) Id of the OIDC credential issuer is all that is required to create a presentation request template. In the application which issues credentials, ie the NationalDrivingLicense, a Razor page was created to view the DID of the OIDC credential issuer.

The DID can be used to create the presentation template. The MATTR documentation is really good here:

https://learn.mattr.global/tutorials/verify/presentation-request-template

A Razor page was created to start this task from the UI. This would normally require authentication as this is an administrator task from the application requesting the verified credentials. The code behind the Razor page takes the DID request parameter and calls the MattrPresentationTemplateService to create the presentation template and present this id a database.

public class CreatePresentationTemplateModel : PageModel { private readonly MattrPresentationTemplateService _mattrVerifyService; public bool CreatingPresentationTemplate { get; set; } = true; public string TemplateId { get; set; } [BindProperty] public PresentationTemplate PresentationTemplate { get; set; } public CreatePresentationTemplateModel(MattrPresentationTemplateService mattrVerifyService) { _mattrVerifyService = mattrVerifyService; } public void OnGet() { PresentationTemplate = new PresentationTemplate(); } public async Task<IActionResult> OnPostAsync() { if (!ModelState.IsValid) { return Page(); } TemplateId = await _mattrVerifyService.CreatePresentationTemplateId(PresentationTemplate.DidId); CreatingPresentationTemplate = false; return Page(); } } public class PresentationTemplate { [Required] public string DidId { get; set; } }

The Razor page html template creates a form to post the request to the server rendered page and displays the templateId after, if the creation was successful.

@page @model BoInsurance.Pages.CreatePresentationTemplateModel <div class="container-fluid"> <div class="row"> <div class="col-sm"> <form method="post"> <div> <div class="form-group"> <label class="control-label">DID ID</label> <input asp-for="PresentationTemplate.DidId" class="form-control" /> <span asp-validation-for="PresentationTemplate.DidId" class="text-danger"></span> </div> <div class="form-group"> @if (Model.CreatingPresentationTemplate) { <input class="form-control" type="submit" readonly="@Model.CreatingPresentationTemplate" value="Create Presentation Template" /> } </div> <div class="form-group"> @if (!Model.CreatingPresentationTemplate) { <div class="alert alert-success"> <strong>Mattr Presentation Template created</strong> </div> } </div> </div> </form> <hr /> <p>When the templateId is created, you can use the template ID to verify</p> </div> <div class="col-sm"> <div> <img src="~/ndl_car_01.png" width="200" alt="Driver License"> <div> <b>Driver Licence templateId from presentation template</b> <hr /> <dl class="row"> <dt class="col-sm-4">templateId</dt> <dd class="col-sm-8"> @Model.TemplateId </dd> </dl> </div> </div> </div> </div> </div>

The MattrPresentationTemplateService is used to create the MATTR presentation template. This class uses the MATTR API and sends a HTTP post request with the DID Id of the OIDC credential issuer and creates a presentation template. The service saves the returned payload to a database and returns the template ID as the result. The template ID is required to verify the verifiable credentials.

The MattrTokenApiService is used to request an API token for the MATTR API using the credential of your MATTR account. This service has a simple token cache and only requests new access tokens when no token exists or the token has expired.

The BoInsuranceDbService service is used to access the SQL database using Entity Framework Core. This provides simple methods to persist or select the data as required.

private readonly IHttpClientFactory _clientFactory; private readonly MattrTokenApiService _mattrTokenApiService; private readonly BoInsuranceDbService _boInsuranceDbService; private readonly MattrConfiguration _mattrConfiguration; public MattrPresentationTemplateService(IHttpClientFactory clientFactory, IOptions<MattrConfiguration> mattrConfiguration, MattrTokenApiService mattrTokenApiService, BoInsuranceDbService boInsuranceDbService) { _clientFactory = clientFactory; _mattrTokenApiService = mattrTokenApiService; _boInsuranceDbService = boInsuranceDbService; _mattrConfiguration = mattrConfiguration.Value; } public async Task<string> CreatePresentationTemplateId(string didId) { // create a new one var v1PresentationTemplateResponse = await CreateMattrPresentationTemplate(didId); // save to db var drivingLicensePresentationTemplate = new DrivingLicensePresentationTemplate { DidId = didId, TemplateId = v1PresentationTemplateResponse.Id, MattrPresentationTemplateReponse = JsonConvert .SerializeObject(v1PresentationTemplateResponse) }; await _boInsuranceDbService .CreateDriverLicensePresentationTemplate(drivingLicensePresentationTemplate); return v1PresentationTemplateResponse.Id; } private async Task<V1_PresentationTemplateResponse> CreateMattrPresentationTemplate(string didId) { HttpClient client = _clientFactory.CreateClient(); var accessToken = await _mattrTokenApiService.GetApiToken(client, "mattrAccessToken"); client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken); client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json"); var v1PresentationTemplateResponse = await CreateMattrPresentationTemplate(client, didId); return v1PresentationTemplateResponse; }

The CreateMattrPresentationTemplate method sends the HTTP Post request like in the MATTR API documentation. Creating the payload for the HTTP post request using the MATTR Open API definitions is a small bit complicated. This could be improved with a better Open API definition. In our use case, we just want to create the default template for the OIDC credential issuer and so just require the DID Id. Most of the other properties are fixed values, see the MATTR API docs for more information.

private async Task<V1_PresentationTemplateResponse> CreateMattrPresentationTemplate( HttpClient client, string didId) { // create presentation, post to presentations templates api // https://learn.mattr.global/tutorials/verify/presentation-request-template var createPresentationsTemplatesUrl = $"https://{_mattrConfiguration.TenantSubdomain}/v1/presentations/templates"; var additionalProperties = new Dictionary<string, object>(); additionalProperties.Add("type", "QueryByExample"); additionalProperties.Add("credentialQuery", new List<CredentialQuery> { new CredentialQuery { Reason = "Please provide your driving license", Required = true, Example = new Example { Context = new List<object>{ "https://schema.org" }, Type = "VerifiableCredential", TrustedIssuer = new List<TrustedIssuer2> { new TrustedIssuer2 { Required = true, Issuer = didId // DID use to create the oidc } } } } }); var payload = new MattrOpenApiClient.V1_CreatePresentationTemplate { Domain = _mattrConfiguration.TenantSubdomain, Name = "certificate-presentation", Query = new List<Query> { new Query { AdditionalProperties = additionalProperties } } }; var payloadJson = JsonConvert.SerializeObject(payload); var uri = new Uri(createPresentationsTemplatesUrl); using (var content = new StringContentWithoutCharset(payloadJson, "application/json")) { var presentationTemplateResponse = await client.PostAsync(uri, content); if (presentationTemplateResponse.StatusCode == System.Net.HttpStatusCode.Created) { var v1PresentationTemplateResponse = JsonConvert .DeserializeObject<MattrOpenApiClient.V1_PresentationTemplateResponse>( await presentationTemplateResponse.Content.ReadAsStringAsync()); return v1PresentationTemplateResponse; } var error = await presentationTemplateResponse.Content.ReadAsStringAsync(); } throw new Exception("whoops something went wrong"); }

The application can be started and the presentation template can be created. The ID is returned back to the UI for the next step.

Verify the verifiable credentials

Now that a template exists to request the verifiable data from the holder of the data which is normally stored in a digital wallet, the verifier application can create and start a verification process. A post request is sent to the MATTR APIs which creates a presentation request using a DID ID and the required template. The application can request the DID from the OIDC credential issuer. The request is signed using the correct key from the DID and the request is published in the UI as a QR Code. A digital wallet is used to scan the code and the user of the wallet can grant consent to share the personal data. The wallet sends a HTTP post request to the callback API. This API handles the request, would validate the data and updates the UI using SignalR to move to the next step of the business process using the verified data.

Step 1 Invoke a presentation request

The InvokePresentationRequest method implements the presentation request. This method requires the DID Id of the OIDC credential issuer which will by used to get the data from the holder of the data. The template ID is also required from the template created above. A challenge is also used to track the verification. The challenge is a random value and is used when the digital wallet calls the API with the verified data. The callback URL is where the data is returned to. This could be unique for every request or anything you want. The payload is created like the docs from the MATTR API defines. The post request is sent to the MATTR API and a V1_CreatePresentationRequestResponse is returned if all is configured correctly.

private async Task<V1_CreatePresentationRequestResponse> InvokePresentationRequest( HttpClient client, string didId, string templateId, string challenge, string callbackUrl) { var createDidUrl = $"https://{_mattrConfiguration.TenantSubdomain}/v1/presentations/requests"; var payload = new MattrOpenApiClient.V1_CreatePresentationRequestRequest { Did = didId, TemplateId = templateId, Challenge = challenge, CallbackUrl = new Uri(callbackUrl), ExpiresTime = MATTR_EPOCH_EXPIRES_TIME_VERIFIY // Epoch time }; var payloadJson = JsonConvert.SerializeObject(payload); var uri = new Uri(createDidUrl); using (var content = new StringContentWithoutCharset(payloadJson, "application/json")) { var response = await client.PostAsync(uri, content); if (response.StatusCode == System.Net.HttpStatusCode.Created) { var v1CreatePresentationRequestResponse = JsonConvert .DeserializeObject<V1_CreatePresentationRequestResponse>( await response.Content.ReadAsStringAsync()); return v1CreatePresentationRequestResponse; } var error = await response.Content.ReadAsStringAsync(); } return null; }

Step 2 Get the OIDC Issuer DID

The RequestDID method uses the MATTR API to get the DID data from the blochchain for the OIDC credential issuer. Only the DID Id is required.

private async Task<V1_GetDidResponse> RequestDID(string didId, HttpClient client) { var requestUrl = $"https://{_mattrConfiguration.TenantSubdomain}/core/v1/dids/{didId}"; var uri = new Uri(requestUrl); var didResponse = await client.GetAsync(uri); if (didResponse.StatusCode == System.Net.HttpStatusCode.OK) { var v1CreateDidResponse = JsonConvert.DeserializeObject<V1_GetDidResponse>( await didResponse.Content.ReadAsStringAsync()); return v1CreateDidResponse; } var error = await didResponse.Content.ReadAsStringAsync(); return null; }

Step 3 Sign the request using correct key and display QR Code

To verify data using a digital wallet, the payload must be signed using the correct key. The SignAndEncodePresentationRequestBody uses the DID payload and the request from the presentation request to create the payload to sign. Creating the payload is a big messy due to the OpenAPI definitions created for the MATTR API. A HTTP post request with the payload returns the signed JWT in a payload in a strange data format so we parse this as a string and manually get the JWT payload.

private async Task<string> SignAndEncodePresentationRequestBody( HttpClient client, V1_GetDidResponse did, V1_CreatePresentationRequestResponse v1CreatePresentationRequestResponse) { var createDidUrl = $"https://{_mattrConfiguration.TenantSubdomain}/v1/messaging/sign"; object didUrlArray; did.DidDocument.AdditionalProperties.TryGetValue("authentication", out didUrlArray); var didUrl = didUrlArray.ToString().Split("\"")[1]; var payload = new MattrOpenApiClient.SignMessageRequest { DidUrl = didUrl, Payload = v1CreatePresentationRequestResponse.Request }; var payloadJson = JsonConvert.SerializeObject(payload); var uri = new Uri(createDidUrl); using (var content = new StringContentWithoutCharset(payloadJson, "application/json")) { var response = await client.PostAsync(uri, content); if (response.StatusCode == System.Net.HttpStatusCode.OK) { var result = await response.Content.ReadAsStringAsync(); return result; } var error = await response.Content.ReadAsStringAsync(); } return null; }

The CreateVerifyCallback method uses the presentation request, the get DID and the sign HTTP post requests to create a URL which can be displayed in a UI. The challenge is created using the RNGCryptoServiceProvider class which creates a random string. The access token to access the API is returned from the client credentials OAuth requests or from the in memory cache. The DrivingLicensePresentationVerify class is persisted to a database and the verify URL is returned so that this could be displayed as a QR Code in the UI.

/// <summary> /// https://learn.mattr.global/tutorials/verify/using-callback/callback-e-to-e /// </summary> /// <param name="callbackBaseUrl"></param> /// <returns></returns> public async Task<(string QrCodeUrl, string ChallengeId)> CreateVerifyCallback(string callbackBaseUrl) { callbackBaseUrl = callbackBaseUrl.Trim(); if (!callbackBaseUrl.EndsWith('/')) { callbackBaseUrl = $"{callbackBaseUrl}/"; } var callbackUrlFull = $"{callbackBaseUrl}{MATTR_CALLBACK_VERIFY_PATH}"; var challenge = GetEncodedRandomString(); HttpClient client = _clientFactory.CreateClient(); var accessToken = await _mattrTokenApiService.GetApiToken(client, "mattrAccessToken"); client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken); client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json"); var template = await _boInsuranceDbService.GetLastDriverLicensePrsentationTemplate(); // Invoke the Presentation Request var invokePresentationResponse = await InvokePresentationRequest( client, template.DidId, template.TemplateId, challenge, callbackUrlFull); // Request DID V1_GetDidResponse did = await RequestDID(template.DidId, client); // Sign and Encode the Presentation Request body var signAndEncodePresentationRequestBodyResponse = await SignAndEncodePresentationRequestBody( client, did, invokePresentationResponse); // fix strange DTO var jws = signAndEncodePresentationRequestBodyResponse.Replace("\"", ""); // save to db // TODO add this back once working var drivingLicensePresentationVerify = new DrivingLicensePresentationVerify { DidId = template.DidId, TemplateId = template.TemplateId, CallbackUrl = callbackUrlFull, Challenge = challenge, InvokePresentationResponse = JsonConvert.SerializeObject(invokePresentationResponse), Did = JsonConvert.SerializeObject(did), SignAndEncodePresentationRequestBody = jws }; await _boInsuranceDbService.CreateDrivingLicensePresentationVerify(drivingLicensePresentationVerify); var qrCodeUrl = $"didcomm://https://{_mattrConfiguration.TenantSubdomain}/?request={jws}"; return (qrCodeUrl, challenge); } private string GetEncodedRandomString() { var base64 = Convert.ToBase64String(GenerateRandomBytes(30)); return HtmlEncoder.Default.Encode(base64); } private byte[] GenerateRandomBytes(int length) { using var randonNumberGen = new RNGCryptoServiceProvider(); var byteArray = new byte[length]; randonNumberGen.GetBytes(byteArray); return byteArray; }

The CreateVerifierDisplayQrCodeModel is the code behind for the Razor page to request a verification and also display the verify QR Code for the digital wallet to scan. The CallbackUrl can be set from the UI so that this is easier for testing. This callback can be any webhook you want or API. To test the application in local development, I used ngrok. The return URL has to match the proxy which tunnels to you PC, once you start. If the API has no public address when debugging, you will not be able to test locally.

public class CreateVerifierDisplayQrCodeModel : PageModel { private readonly MattrCredentialVerifyCallbackService _mattrCredentialVerifyCallbackService; public bool CreatingVerifier { get; set; } = true; public string QrCodeUrl { get; set; } [BindProperty] public string ChallengeId { get; set; } [BindProperty] public CreateVerifierDisplayQrCodeCallbackUrl CallbackUrlDto { get; set; } public CreateVerifierDisplayQrCodeModel(MattrCredentialVerifyCallbackService mattrCredentialVerifyCallbackService) { _mattrCredentialVerifyCallbackService = mattrCredentialVerifyCallbackService; } public void OnGet() { CallbackUrlDto = new CreateVerifierDisplayQrCodeCallbackUrl(); CallbackUrlDto.CallbackUrl = $"https://{HttpContext.Request.Host.Value}"; } public async Task<IActionResult> OnPostAsync() { if (!ModelState.IsValid) { return Page(); } var result = await _mattrCredentialVerifyCallbackService .CreateVerifyCallback(CallbackUrlDto.CallbackUrl); CreatingVerifier = false; QrCodeUrl = result.QrCodeUrl; ChallengeId = result.ChallengeId; return Page(); } } public class CreateVerifierDisplayQrCodeCallbackUrl { [Required] public string CallbackUrl { get; set; } }

The html or template part of the Razor page displays the QR Code from a successful post request. You can set any URL for the callback in the form request. This is really just for testing.

@page @model BoInsurance.Pages.CreateVerifierDisplayQrCodeModel <div class="container-fluid"> <div class="row"> <div class="col-sm"> <form method="post"> <div> <div class="form-group"> <label class="control-label">Callback base URL (ngrok in debug...)</label> <input asp-for="CallbackUrlDto.CallbackUrl" class="form-control" /> <span asp-validation-for="CallbackUrlDto.CallbackUrl" class="text-danger"></span> </div> <div class="form-group"> @if (Model.CreatingVerifier) { <input class="form-control" type="submit" readonly="@Model.CreatingVerifier" value="Create Verification" /> } </div> <div class="form-group"> @if (!Model.CreatingVerifier) { <div class="alert alert-success"> <strong>Ready to verify</strong> </div> } </div> </div> </form> <hr /> <p>When the verification is created, you can scan the QR Code to verify</p> </div> <div class="col-sm"> <div> <img src="~/ndl_car_01.png" width="200" alt="Driver License"> </div> </div> </div> <div class="row"> <div class="col-sm"> <div class="qr" id="qrCode"></div> <input asp-for="ChallengeId" hidden/> </div> </div> </div> @section scripts { <script src="~/js/qrcode.min.js"></script> <script type="text/javascript"> new QRCode(document.getElementById("qrCode"), { text: "@Html.Raw(Model.QrCodeUrl)", width: 400, height: 400, correctLevel: QRCode.CorrectLevel.M }); $(document).ready(() => { }); var connection = new signalR.HubConnectionBuilder().withUrl("/mattrVerifiedSuccessHub").build(); connection.on("MattrCallbackSuccess", function (challengeId) { console.log("received verification:" + challengeId); window.location.href = "/VerifiedUser?challengeid=" + challengeId; }); connection.start().then(function () { //console.log(connection.connectionId); const challengeId = $("#ChallengeId").val(); if (challengeId) { console.log(challengeId); // join message connection.invoke("AddChallenge", challengeId, connection.connectionId).catch(function (err) { return console.error(err.toString()); }); } }).catch(function (err) { return console.error(err.toString()); }); </script> }

Step 4 Implement the Callback and update the UI using SignalR

After a successful verification in the digital wallet, the wallet sends the verified credentials to the API defined in the presentation request. The API handling this needs to update the correct client UI and continue the business process using the verified data. We use SignalR for this with a single client to client connection. The Signal connections for each connection is associated with a challenge ID, the same Id we used to create the presentation request. Using this, only the correct client will be notified and not all clients broadcasted. The DrivingLicenseCallback takes the body with is specific for the credentials you issued. This is always depending on what you request. The data is saved to a database and the client is informed to continue. We send a message directly to the correct client using the connectionId of the SignalR session created for this challenge.

[ApiController] [Route("api/[controller]")] public class VerificationController : Controller { private readonly BoInsuranceDbService _boInsuranceDbService; private readonly IHubContext<MattrVerifiedSuccessHub> _hubContext; public VerificationController(BoInsuranceDbService boInsuranceDbService, IHubContext<MattrVerifiedSuccessHub> hubContext) { _hubContext = hubContext; _boInsuranceDbService = boInsuranceDbService; } /// <summary> /// { /// "presentationType": "QueryByExample", /// "challengeId": "GW8FGpP6jhFrl37yQZIM6w", /// "claims": { /// "id": "did:key:z6MkfxQU7dy8eKxyHpG267FV23agZQu9zmokd8BprepfHALi", /// "name": "Chris", /// "firstName": "Shin", /// "licenseType": "Certificate Name", /// "dateOfBirth": "some data", /// "licenseIssuedAt": "dda" /// }, /// "verified": true, /// "holder": "did:key:z6MkgmEkNM32vyFeMXcQA7AfQDznu47qHCZpy2AYH2Dtdu1d" /// } /// </summary> /// <param name="body"></param> /// <returns></returns> [HttpPost] [Route("[action]")] public async Task<IActionResult> DrivingLicenseCallback([FromBody] VerifiedDriverLicense body) { string connectionId; var found = MattrVerifiedSuccessHub.Challenges .TryGetValue(body.ChallengeId, out connectionId); // test Signalr //await _hubContext.Clients.Client(connectionId).SendAsync("MattrCallbackSuccess", $"{body.ChallengeId}"); //return Ok(); var exists = await _boInsuranceDbService.ChallengeExists(body.ChallengeId); if (exists) { await _boInsuranceDbService.PersistVerification(body); if (found) { //$"/VerifiedUser?challengeid={body.ChallengeId}" await _hubContext.Clients .Client(connectionId) .SendAsync("MattrCallbackSuccess", $"{body.ChallengeId}"); } return Ok(); } return BadRequest("unknown verify request"); } }

The SignalR server is configured in the Startup class of the ASP.NET Core application. The path for the hub is defined in the MapHub method.

public void ConfigureServices(IServiceCollection services) { // ... services.AddRazorPages(); services.AddSignalR(); services.AddControllers(); } public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { // ... app.UseEndpoints(endpoints => { endpoints.MapRazorPages(); endpoints.MapHub<MattrVerifiedSuccessHub>("/mattrVerifiedSuccessHub"); endpoints.MapControllers(); }); }

The Hub implementation requires only one fixed method. The AddChallenge method takes the challenge Id and adds this the an in-memory cache. The controller implemented for the callbacks uses this ConcurrentDictionary to find the correct connectionId which is mapped to the challenges form the verification.

public class MattrVerifiedSuccessHub : Hub { /// <summary> /// This should be replaced with a cache which expires or something /// </summary> public static readonly ConcurrentDictionary<string, string> Challenges = new ConcurrentDictionary<string, string>(); public void AddChallenge(string challengeId, string connnectionId) { Challenges.TryAdd(challengeId, connnectionId); } }

The Javascript SignalR client in the browser connects to the SignalR server and registers the connectionId with the challenge ID used for the verification of the verifiable credentials from the holder of the digital wallet. If a client gets a message from that a verify has completed successfully and the callback has been called, it will redirect to the verified page. The client listens to the MattrCallbackSuccess for messages. These messages are sent from the callback controller directly.

<script type="text/javascript"> var connection = new signalR.HubConnectionBuilder() .withUrl("/mattrVerifiedSuccessHub").build(); connection.on("MattrCallbackSuccess", function (challengeId) { console.log("received verification:" + challengeId); window.location.href = "/VerifiedUser?challengeid=" + challengeId; }); connection.start().then(function () { //console.log(connection.connectionId); const challengeId = $("#ChallengeId").val(); if (challengeId) { console.log(challengeId); // join message connection.invoke("AddChallenge", challengeId, connection.connectionId).catch(function (err) { return console.error(err.toString()); }); } }).catch(function (err) { return console.error(err.toString()); }); </script>

The VerifiedUserModel Razor page displays the data and the business process can continue using the verified data.

public class VerifiedUserModel : PageModel { private readonly BoInsuranceDbService _boInsuranceDbService; public VerifiedUserModel(BoInsuranceDbService boInsuranceDbService) { _boInsuranceDbService = boInsuranceDbService; } public string ChallengeId { get; set; } public DriverLicenseClaimsDto VerifiedDriverLicenseClaims { get; private set; } public async Task OnGetAsync(string challengeId) { // user query param to get challenge id and display data if (challengeId != null) { var verifiedDriverLicenseUser = await _boInsuranceDbService.GetVerifiedUser(challengeId); VerifiedDriverLicenseClaims = new DriverLicenseClaimsDto { DateOfBirth = verifiedDriverLicenseUser.DateOfBirth, Name = verifiedDriverLicenseUser.Name, LicenseType = verifiedDriverLicenseUser.LicenseType, FirstName = verifiedDriverLicenseUser.FirstName, LicenseIssuedAt = verifiedDriverLicenseUser.LicenseIssuedAt }; } } } public class DriverLicenseClaimsDto { public string Name { get; set; } public string FirstName { get; set; } public string LicenseType { get; set; } public string DateOfBirth { get; set; } public string LicenseIssuedAt { get; set; } }

Running the verifier

To test the BoInsurance application locally, which is the verifier application, ngrok is used so that we have a public address for the callback. I install ngrok using npm. Without a license, you can only run your application in http.

npm install -g ngrok

Run the ngrok from the command line using the the URL of the application. I start the ASP.NET Core application at localhost port 5000.

ngrok http localhost:5000

You should be able to copied the ngrok URL and use this in the browser to test the verification.

Once running, a verification can be created and you can scan the QR Code with your digital wallet. Once you grant access to your data, the data is sent to the callback API and the UI will be redirected to the success page.

Notes

MATTR APIs work really well and support some of the flows for digital identities. I plan to try out the zero proof flow next. It is only possible to create verifiable credentials from data from your identity provider using the id_token. To issue credentials, you have to implement your own identity provider and cannot use business data from your application. If you have full control like with Openiddict, IdenityServer4 or Auth0, this is no problem, just more complicated to implement. If you do not control the data in your identity provider, you would need to create a second identity provider to issue credentials. This is part of your business logic then and not just an identity provider. This will always be a problem is using Azure AD or IDPs from large, medium size companies. The quality of the verifiable credentials also depend on how good the OIDC credential issuers are implemented as these are still central databases for these credentials and are still open to all the problems we have today. Decentralized identities have to potential to solve many problems but still have many unsolved problems.

Links

https://mattr.global/

https://learn.mattr.global/tutorials/verify/using-callback/callback-e-to-e

https://mattr.global/get-started/

https://learn.mattr.global/

https://keybase.io/

https://learn.mattr.global/tutorials/dids/did-key

https://gunnarpeipman.com/httpclient-remove-charset/

https://auth0.com/

Wednesday, 05. May 2021

Nader Helmy

IIW32: BBS+ and beyond

The Internet Identity Workshop continues to be a central nucleus for thoughtful discussion and development of all things related to digital identity. The most recent workshop, which was held in mid-April, was no exception. Despite the lack of in-person interaction due to the ongoing global pandemic, this IIW was as lively as ever, bringing together a diverse set of stakeholders from across the glo

The Internet Identity Workshop continues to be a central nucleus for thoughtful discussion and development of all things related to digital identity. The most recent workshop, which was held in mid-April, was no exception. Despite the lack of in-person interaction due to the ongoing global pandemic, this IIW was as lively as ever, bringing together a diverse set of stakeholders from across the globe to share experiences, swap perspectives, and engage in healthy debates.

One common theme this year was the continued development and adoption of BBS+ signatures, a type of multi-message cryptographic digital signature that enables selective disclosure of verifiable credentials. We first introduced this technology at IIW30 in April 2020, and have been inspired and delighted by the community’s embrace and contribution to this effort across the board. In the year since, progress has been made in a variety of areas, from standards-level support to independent implementations and advanced feature support.

We thought we’d take a moment to round up some of the significant developments surrounding BBS+ signatures and highlight a few of the top items to pay attention to going forward.

Over the past few months, the linked data proofs reference implementation of BBS+ published a new release that introduces a variety of improvements in efficiency and security, including formal alignment to the W3C CCG Security Vocab v3 definitions. In addition, support for JSON-LD BBS+ signatures was added to the VC HTTP API, making it possible to test this functionality in an interoperable way with other vendors participating in an open environment.

An important element in enabling BBS+ signatures is using what’s known as a pairing-friendly curve; for our purposes we use BLS12–381. We have seen some promising signs of adoption for this key pair, with multiple Decentralized Identifier (DID) methods — both did:indy from Hyperledger and did:ion from DIF — indicating they intend to add or already have support for these keys, allowing BBS+ signatures to be issued across a variety of decentralized networks and ecosystems. This development is possible due to the fact that BBS+ signatures is a ledger-independent approach to selective disclosure, effectively no custom logic or bespoke infrastructure is needed for these digital signatures to be created, used and understood.

In addition, the Hyperledger Aries project has been hard at work developing interoperable and ledger-agnostic capabilities in open source. The method used to track interop targets within the cohort and ultimately measure conformance against Aries standards is what’s known as an Aries Interop Profile (AIP). A major upcoming update to AIP will add support for additional DID methods, key types and credential formats, as well as introducing Aries support for JSON-LD BBS+ signatures as part of AIP 2.0. This will allow Aries-driven credential issuance and presentation protocols to work natively with BBS+ credentials, making that functionality broadly available for those in the Aries community and beyond.

There have also been a number of exciting developments when it comes to independent implementations of BBS+ signatures. Animo Solutions has recently implemented JSON-LD BBS+ signatures support into the popular open-source codebase Hyperledger Aries Cloud Agent Python (ACA-Py). In another independent effort, Trinsic has contributed an implementation of JSON-LD BBS+ credentials which they have demonstrated to be working in tandem with DIDComm v2, a secure messaging protocol based on DIDs. Implementations such as these help to demonstrate that open standards are transparent, can be understood and verified independently, and can be implemented with separate languages and tech stacks. They also set the groundwork for demonstrating real testing-driven interoperability via mechanisms such as the VC HTTP API and AIP 2.0. We are continuously looking to improve the documentation of these specs and standards so that their implications and nuances can be more broadly understood by builders and developers looking to engage with the technology.

On the cryptographic side of things, progress is also being made in hardening the existing BBS+ specification as well as expanding BBS+ to support more advanced privacy-preserving features. A significant development in this area is the work of cryptographer Michael Lodder who has been actively conducting research on an enhanced credential revocation mechanism using cryptographic accumulators with BBS+. This approach presents a promising alternative to existing solutions that allow authoritative issuers to update the status of issued credentials without compromising the privacy of the credential holder or subject who may be presenting the credential. We see this as another application of BBS+ signatures in the context of verifiable credentials that carries a lot of potential in pushing this technology to an even more robust state.

There was also initial discussion and tacit agreement to create a new cryptography-focused working group at Decentralized Identity Foundation. As the new WG drafts its charter, the first work item of this group will be the BBS+ Signatures spec which defines the cryptographic scheme known as BBS+ agnostic of its application in areas such as linked data signatures or verifiable credentials. In the future, this WG will likely evolve to include other crypto-related work items from the community.

This is just the tip of the iceberg when it comes to the momentum and development building around this technology in the community. We couldn’t be more excited about the future of BBS+ signatures, especially as we gear up to tackle the next set of hard problems in this area including privacy-preserving subject authentication and revocation using cryptographic accumulators. If you’re interested we encourage you to get involved, either by contributing to the Linked Data Proofs specification, checking out our reference implementations, or participating in the new WG at DIF, to name but a few of the many ways to engage with this work. We look forward to doing this retrospective at many IIWs to come, documenting the ever-growing community that continues to champion this technology in dynamic and interesting ways.

IIW32: BBS+ and beyond was originally published in MATTR on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tuesday, 04. May 2021

Justin Richer

Signing HTTP Messages

There’s a new draft in the HTTP working group that deals with signing HTTP messages of all types. Why is it here, and what does that give us? HTTP is irrefutably a fundamental building block of most of today’s software systems. Yet security and identity need to be layered alongside HTTP. The most common of these is simply running the HTTP protocol over an encrypted socket using TLS, resultin

There’s a new draft in the HTTP working group that deals with signing HTTP messages of all types. Why is it here, and what does that give us?

HTTP is irrefutably a fundamental building block of most of today’s software systems. Yet security and identity need to be layered alongside HTTP. The most common of these is simply running the HTTP protocol over an encrypted socket using TLS, resulting in HTTPS. While this is a powerful and important security component, TLS works only my protecting the stream of bits in transit. It does not allow for message-level and application-level security operations. But what if we could sign the messages themselves?

While it is possible to wrap the body of a request in a cryptographic envelope like JOSE or XML DSig, such approaches force developers to ignore most of the power and flexibility of HTTP, reducing it to a dumb transport layer. In order to sign a message but keep using HTTP as it stands, with all the verbs and headers and content types that it gives us, we will need a scheme that allows us to add a detached signature to the HTTP message. The cryptographic elements to the message can then be generated and validated separately from the request itself, providing a layered approach.

There have been numerous attempts at creating detached signature methods for HTTP over the years, one of the most famous being the Cavage draft which itself started as a community-facing version of Amazon’s SIGv4 method used within AWS. There were several other efforts, and all of them incompatible with each other in one way or another. To address this, the HTTP Working Group in the IETF stepped up and took on the effort of creating an RFC-track standard for HTTP message signatures that could be used across the variety of use cases.

As of the writing of this post, the specification is at version 04. While it’s not finished yet, it’s recently become a bit more stable and so it’s worth looking at it in greater depth.

Normalizing HTTP

As it turns out, the hardest part of signing HTTP messages isn’t the signing, it’s the HTTP. HTTP is a messy set of specifications, with pieces that have been built up by many authors over many years in ways that aren’t always that consistent. A recent move towards consistency has been the adoption of Structured Field Values for HTTP. In short, structured fields allow HTTP headers to house simple, non-recursive data structures with unambiguous parsing and deterministic serialization. These aspects made it perfect for use within the HTTP message signatures specification.

Previous efforts at HTTP message signing concentrated on creating a signature around HTTP headers, and the current draft is no exception in allowing that. On top of that, the current draft also allows for the definition of specialty fields that contain other pieces of constructed information not found in the headers themselves. These covered components are identified and combined with each other into a signature input string. To this string is added a field that includes all of the input parameters to this signature. For example, let’s say we want to sign parts of this HTTP request:

POST /foo?param=value&pet=dog HTTP/1.1
Host: example.com
Date: Tue, 20 Apr 2021 02:07:55 GMT
Content-Type: application/json
Digest: SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
Content-Length: 18

{"hello": "world"}

We choose the components we want to sign, including the target of the request and a subset of the available headers, and create the following signature input string:

@request-target": post /foo?param=value&pet=dog
"host": example.com
"date": Tue, 20 Apr 2021 02:07:55 GMT
"content-type": application/json
"digest": SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
"content-length": 18
"@signature-params": ("@request-target" "host" "date" "content-type" "digest" "content-length");created=1618884475;keyid="test-key-rsa-pss"

With a given HTTP message and a set of input parameters determining which parts of the message are covered with a signature, any party can re-generate this string with a reasonable level of success. Unsigned headers can be added to the message by intermediaries without invalidating the signature, and it’s even possible for an intermediary to add its own signature to the message on the way through — but we’ll get more into that advanced use case in a future post. The result of this is that the signer and verifier will re-create this signature input string independently of each other.

Now that we have a normalized string to sign, how do we actually sign it?

Signing and Verifying Content

Once we have the string, it’s a relatively straightforward matter of applying a key and signature function to the string. Any signature method that takes in and bunch of bytes and spits out a different set of bytes is technically feasible here.

How do the signer and verifier know which algorithm to use on a given method? It turns out that different deployments have drastically different needs in this regard. As a consequence, this is an aspect that is application specific by the specification, with several common methods called out:

The signer and verifier can both be configured to expect only a specific algorithm, or have that algorithm identified by some aspect external to the protocol. The signer and verifier can identify the key used to do the signing and figure out the signature algorithm based on that. If an application’s using JSON Web Keys, the alg field of the key provides an easy way to identify a signing mechanism. If the signer and verifier need to signal the algorithm dynamically at runtime, there is an alg field in the signature parameter set itself that points to a new registry. And if two or more of these methods are applicable to a given message, the answers all have to match, otherwise something fishy is going on and the signature is invalidated.

Given the above signature input string and an RSA-PSS signing method, we end up with the following Base64-encoded bytes as the signature output:

NtIKWuXjr4SBEXj97gbick4O95ff378I0CZOa2VnIeEXZ1itzAdqTpSvG91XYrq5CfxCmk8zz1Zg7ZGYD+ngJyVn805r73rh2eFCPO+ZXDs45Is/Ex8srzGC9sfVZfqeEfApRFFe5yXDmANVUwzFWCEnGM6+SJVmWl1/jyEn45qA6Hw+ZDHbrbp6qvD4N0S92jlPyVVEh/SmCwnkeNiBgnbt+E0K5wCFNHPbo4X1Tj406W+bTtnKzaoKxBWKW8aIQ7rg92zqE1oqBRjqtRi5/Q6P5ZYYGGINKzNyV3UjZtxeZNnNJ+MAnWS0mofFqcZHVgSU/1wUzP7MhzOKLca1Yg==

This gives us a signed object, and now we need to put that into our HTTP message.

Sending Signatures in Messages

The HTTP message signature specification defines two new headers to carry the signature, Signature and Signature-Input. Both of these use the Dictionary construct from the HTTP Structured Field Values standard to carry a named signature.

But first, why two headers? This construct allows us to easily separate the metadata about the signature — how it was made — from the signature value itself. This separation makes parsing simpler and also allows the HTTP message signatures specification to support multiple independent signatures on a given message.

The Signature-Input header contains all the parameters that went into the creation of the signature, including the list of covered content, identifiers for the key and algorithm, and items like timestamps or other application-specific flags. In fact, this is the same value used as the last line of the signature input string, and so its values are always covered by the signature. The Signature header contains the value of the signature itself as a byte array, encoded in Base64. The signer chooses a name for the signature object and adds both items to the headers. The name has no semantic impact, it just needs to be unique within a given request.

Let’s say this signature is named sig1. The signer adds both headers to the request above, resulting in the following signed request.

POST /foo?param=value&pet=dog HTTP/1.1
Host: example.com
Date: Tue, 20 Apr 2021 02:07:55 GMT
Content-Type: application/json
Digest: SHA-256=X48E9qOokqqrvdts8nOJRJN3OWDUoyWxBf7kbu9DBPE=
Content-Length: 18
Signature-Input: sig1=("host" "date" "content-type");created=1618884475;keyid="test-key-rsa-pss"
Signature: sig1=:NtIKWuXjr4SBEXj97gbick4O95ff378I0CZOa2VnIeEXZ1itzAdqTpSvG91XYrq5CfxCmk8zz1Zg7ZGYD+ngJyVn805r73rh2eFCPO+ZXDs45Is/Ex8srzGC9sfVZfqeEfApRFFe5yXDmANVUwzFWCEnGM6+SJVmWl1/jyEn45qA6Hw+ZDHbrbp6qvD4N0S92jlPyVVEh/SmCwnkeNiBgnbt+E0K5wCFNHPbo4X1Tj406W+bTtnKzaoKxBWKW8aIQ7rg92zqE1oqBRjqtRi5/Q6P5ZYYGGINKzNyV3UjZtxeZNnNJ+MAnWS0mofFqcZHVgSU/1wUzP7MhzOKLca1Yg==: {"hello": "world"}

Note that none of the other headers or aspects of the message are modified by the signature process.

The verifier parses both headers, re-creates the signature input string from the request, and verifies the signature value using the identified key and algorithm. But how does the verifier know that this signature is sufficient for this request, and how does the signer know what to sign in the first place?

Applying the Message Signature Specification

As discussed above, the signer and verifier need to have a way of figuring out which algorithm and keys are appropriate for a given signed message. In many deployments, this information can be gleaned through context and configuration. For example, a key derivation algorithm based on the tenant identifier in the URL can be used to dereference the key needed for a given call. Or an application identifier passed in the body could point to a record giving both the expected algorithm and allowable key material.

In addition to defining a predictable way to determine this, an application of the HTTP message signatures specification also needs to define which parts of the message need to be signed. For example, and API might have very different behaviors based on a Content-Type header but not really care about the Content-Encoding. A security protocol like OAuth or GNAP would require signing the Authorization header that contains the access token as well as the @request-target specialty field.

The HTTP protocol is also designed to allow interception and proxy of requests and responses, with intermediaries fully allowed to alter the message in certain ways. Applications that need to account for such intermediaries can be picky about which headers and components are signed, allowing the signature to survive expected message modifications but protecting against unanticipated changes in transit.

This fundamentally means that no signature method will ever be perfect for all messages — but that’s ok. The HTTP message signature draft instead leans on flexibility, allowing applications to define how best to apply the signature methods to achieve the security needed.

Building the Standard

The HTTP message signatures specification is still a long way from being done. It’s taken in a number of different inputs and many years of collective community experience, and that initially resulted in some major churn in the specification’s syntax and structure. As of version 04 though, the major surgery seems to be behind us. While there will inevitably be some changes to the parameters, names, and possibly even structures, the core of this is pretty solid. It’s time to start implementing it and testing it out with applications of all stripes, and I invite all of you to join me in doing just that.


MyDigitalFootprint

In leadership, why is recognising paradox critically important?

Source: Wendy Smith  https://www.learninginnovationslab.org/guest-faculty/ The importance of creating or seeing a paradox is that you can understand that the data and facts being presented to you can lead to the recommendation or conclusion being offered, but equally that the same data and facts can equally lead to a different conclusion.   Our problem is that we are not very

Source: Wendy Smith  https://www.learninginnovationslab.org/guest-faculty/


The importance of creating or seeing a paradox is that you can understand that the data and facts being presented to you can lead to the recommendation or conclusion being offered, but equally that the same data and facts can equally lead to a different conclusion.  

Our problem is that we are not very good at finding flaws in our own arguments, if for no other reason than they support our incentives and beliefs. We tend to take it personally when someone attacks our logic, beliefs or method, even if they are searching for the paradox. Equally, the person you are about to question reacts just like you do.  

Searching for the paradox allows you to see the jumps, assumptions and framing in the logic being presented, which lays bare how our thinking and decisions are being manipulated.  Often it turns out, others are blinded to see one conclusion, and as a leader and executive, your role is to explore and question the flow.  

Logical flow decisions often create paradoxes because of an invalid argument, but they are nevertheless valuable in creating a narrative. We see this as a statement or proposition which, despite sound (or apparently sound) reasoning from acceptable premises, leads to a conclusion that seems logically unacceptable or self-contradictory.  Finding a paradox in non-logical flow decisions reveals errors in definitions that were assumed to be rigorous. Equally,  a paradox can be seen when a seemingly absurd or contradictory statement or proposition, which when investigated, proves to be well-founded or true.   What is evident is the need for critical thinking, questions and sensitivity. 

Just because an idea pops into your head during a presentation doesn’t mean it’s true or reasonable. Idea bias is a new skinny belief you have just created, leading you to poor decision making as you have been framed. Often in a meeting, the framing is such that the presenter has set up a story or analogy which you believe in and fail to create new questions about (idea bias); as a way to make the logic jumps needed to justify a story.  If you cannot see the paradox, you are in their model, which means you are unlikely to make an unbiased decision.  If you can see the paradox you have mastered critical thinking and can use tools to ensure you make decisions that lead to outcomes that you want. 

 If you cannot see the paradox, you are in a model.


Decision-making framing itself create paradox’s for us 

Prevention paradox: For one person to benefit, many people have to change their behaviour — even though they receive no benefit or even suffer, from the change.  An assumption about the adoption of a product.

Decision-making paradox: Picking “the best decision-making method” is a decision problem in itself. Can the tool pick the best tool?  What has your process already framed as a decision method?

Abilene paradox: Making a decision based on what you think others want to do and not on what they actually want to do.  Everybody decides to do something that nobody really wants to do, but only what they thought everybody else wanted to do.  Do we have the agency to make an individual choice in the setting we have?

Inventor’s paradox: It is easier to solve a more general problem that covers the specifics of the sought-after solution.  Have we actually solved the problem?

Willpower paradox: Those who kept their minds open were more goal-directed and more motivated than those who declared their objective to themselves.

Buridan’s ass: Making a rational choice between two outcomes of equal value creates the longest delay in decision making (thanks, Yael).  Better known as Fredkin’s paradox: The more similar two choices are, the more time a decision-making agent spends on deciding.

Navigation paradox: Increased navigational precision may result in increased collision risk.

The paradox of tolerance: Should one tolerate intolerance if intolerance would destroy the possibility of tolerance?

Prevention paradox: For one person to benefit, many people have to change their behaviour — even though they receive no benefit or even suffer, from the change.

Willpower paradox: Those who kept their minds open were more goal-directed and more motivated than those who declared their objective to themselves.

Rule-following paradox: Even though rules are intended to determine actions, “no course of action could be determined by a rule because any course of action can be made out, to accord with the rule.”


A growing list of paradoxes that can help develop critical thinking can be found here.  I am exploring Paradox as I expand on my thinking at www.peakparadox.com 



Voidstar: blog

Dead Lies Dreaming (Laundry Files, 10) by Charles Stross

[from: Librarything]

[from: Librarything]

Harrow the Ninth (The Locked Tomb Trilogy, 2) by Tamsyn Muir

[from: Librarything]

[from: Librarything]

False Value (Rivers of London) by Ben Aaronovitch

[from: Librarything]

[from: Librarything]

Attack Surface by Cory Doctorow

[from: Librarything]

[from: Librarything]

Robot Artists & Black Swans: The Italian Fantascienza Stories by Bruce Sterling

[from: Librarything]

[from: Librarything]

What Abigail Did That Summer by Ben Aaronovitch

[from: Librarything]

[from: Librarything]

Monday, 03. May 2021

Damien Bod

Create an OIDC credential Issuer with MATTR and ASP.NET Core

This article shows how to create and issue verifiable credentials using MATTR and an ASP.NET Core. The ASP.NET Core application allows an admin user to create an OIDC credential issuer using the MATTR service. The credentials are displayed in an ASP.NET Core Razor Page web UI as a QR code for the users of the […]

This article shows how to create and issue verifiable credentials using MATTR and an ASP.NET Core. The ASP.NET Core application allows an admin user to create an OIDC credential issuer using the MATTR service. The credentials are displayed in an ASP.NET Core Razor Page web UI as a QR code for the users of the application. The user can use a digital wallet form MATTR to scan the QR code, authenticate against an Auth0 identity provider configured for this flow and use the claims from the id token to add the verified credential to the digital wallet. In a follow up post, a second application will then use the verified credentials to allow access to a second business process.

Code: https://github.com/swiss-ssi-group/MattrGlobalAspNetCore

Blogs in the series

Getting started with Self Sovereign Identity SSI Create an OIDC credential Issuer with MATTR and ASP.NET Core Present and Verify Verifiable Credentials in ASP.NET Core using Decentralized Identities and MATTR Verify vaccination data using Zero Knowledge Proofs with ASP.NET Core and MATTR

Setup

The solutions involves an MATTR API which handles all the blockchain identity logic. An ASP.NET Core application is used to create the digital identity and the OIDC credential issuer using the MATTR APIs and also present this as a QR code which can be scanned. An identity provider is required to add the credential properties to the id token. The properties in a verified credential are issued using the claims values from the id token so a specific identity provider is required with every credential issuer using this technic. Part of the business of this solution is adding business claims to the identity provider. A MATTR digital wallet is required to scan the QR code, authenticate against the OIDC provider which in our case is Auth0 and then store the verified credentials to the wallet for later use.

MATTR Setup

You need to register with MATTR and create a new account. MATTR will issue you access to your sandbox domain and you will get access data from them plus a link to support.

Once setup, use the OIDC Bridge tutorial to implement the flow used in this demo. The docs are really good but you need to follow the docs exactly.

https://learn.mattr.global/tutorials/issue/oidc-bridge/issue-oidc

Auth0 Setup

A standard trusted web application which supports the code flow is required so that the MATTR digital wallet can authenticate using the identity provider and use the id token values from the claims which are required in the credential. It is important to create a new application which is only used for this because the client secret is required when creating the OIDC credential issuer and is shared with the MATTR platform. It would probably be better to use certificates instead of a shared secret which is persisted in different databases. We also use a second Auth0 application configuration to sign into the web application but this is not required to issue credentials.

In Auth0, rules are used to extend the id token claims. You need to add your claims as required by the MATTR API and your business logic for the credentials you wish to issue.

function (user, context, callback) { const namespace = 'https://--your-tenant--.vii.mattr.global/'; context.idToken[namespace + 'license_issued_at'] = user.user_metadata.license_issued_at; context.idToken[namespace + 'license_type'] = user.user_metadata.license_type; context.idToken[namespace + 'name'] = user.user_metadata.name; context.idToken[namespace + 'first_name'] = user.user_metadata.first_name; context.idToken[namespace + 'date_of_birth'] = user.user_metadata.date_of_birth; callback(null, user, context); }

For every user (holder) who should be able to create verifiable credentials, you must add the credential data to the user profile. This is part of the business process with this flow. If you were to implement this for a real application with lots of users, it would probably be better to integrate the identity provider into the solution issuing the credentials and add a UI for editing the user profile data which is used in the credentials. This would be really easy using ASP.NET Core Identity and for example OpenIddict or IdentityServer4. It is important that the user cannot edit this data. This logic is part of the credential issuer logic and not part of the user profile.

After creating a new MATTR OIDC credential issuer, the callback URL needs to be added to the Open ID connect code flow client used for the digital wallet sign in.

Add the URL to the Allowed Callback URLs in the settings of your Auth0 application configuration for the digital wallet.

Implementing the OpenID Connect credentials Issuer application

The ASP.NET Core application is used to create new OIDC credential issuers and also display the QR code for these so that the verifiable credential can be loaded to the digital wallet. The application requires secrets. The data is stored to a database, so that any credential can be added to a wallet at a later date and also so that you can find the credentials you created. The MattrConfiguration is the data and the secrets you received from MATTR for you account access to the API. The Auth0 configuration is the data required to sign in to the application. The Auth0Wallet configuration is the data required to create the OIDC credential issuer so that the digital wallet can authenticate the identity using the Auth0 application. This data is stored in the user secrets during development.

{ // use user secrets "ConnectionStrings": { "DefaultConnection": "--your-connection-string--" }, "MattrConfiguration": { "Audience": "https://vii.mattr.global", "ClientId": "--your-client-id--", "ClientSecret": "--your-client-secret--", "TenantId": "--your-tenant--", "TenantSubdomain": "--your-tenant-sub-domain--", "Url": "http://mattr-prod.au.auth0.com/oauth/token" }, "Auth0": { "Domain": "--your-auth0-domain", "ClientId": "--your--auth0-client-id--", "ClientSecret": "--your-auth0-client-secret--", } "Auth0Wallet": { "Domain": "--your-auth0-wallet-domain", "ClientId": "--your--auth0-wallet-client-id--", "ClientSecret": "--your-auth0-wallet-client-secret--", } }

Accessing the MATTR APIs

The MattrConfiguration DTO is used to fetch the MATTR account data for the API access and to use in the application.

public class MattrConfiguration { public string Audience { get; set; } public string ClientId { get; set; } public string ClientSecret { get; set; } public string TenantId { get; set; } public string TenantSubdomain { get; set; } public string Url { get; set; } }

The MattrTokenApiService is used to acquire an access token and used for the MATTR API access. The token is stored to a cache and only fetched if the old one has expired or is not available.

public class MattrTokenApiService { private readonly ILogger<MattrTokenApiService> _logger; private readonly MattrConfiguration _mattrConfiguration; private static readonly Object _lock = new Object(); private IDistributedCache _cache; private const int cacheExpirationInDays = 1; private class AccessTokenResult { public string AcessToken { get; set; } = string.Empty; public DateTime ExpiresIn { get; set; } } private class AccessTokenItem { public string access_token { get; set; } = string.Empty; public int expires_in { get; set; } public string token_type { get; set; } public string scope { get; set; } } private class MattrCrendentials { public string audience { get; set; } public string client_id { get; set; } public string client_secret { get; set; } public string grant_type { get; set; } = "client_credentials"; } public MattrTokenApiService( IOptions<MattrConfiguration> mattrConfiguration, IHttpClientFactory httpClientFactory, ILoggerFactory loggerFactory, IDistributedCache cache) { _mattrConfiguration = mattrConfiguration.Value; _logger = loggerFactory.CreateLogger<MattrTokenApiService>(); _cache = cache; } public async Task<string> GetApiToken(HttpClient client, string api_name) { var accessToken = GetFromCache(api_name); if (accessToken != null) { if (accessToken.ExpiresIn > DateTime.UtcNow) { return accessToken.AcessToken; } else { // remove => NOT Needed for this cache type } } _logger.LogDebug($"GetApiToken new from oauth server for {api_name}"); // add var newAccessToken = await GetApiTokenClient(client); AddToCache(api_name, newAccessToken); return newAccessToken.AcessToken; } private async Task<AccessTokenResult> GetApiTokenClient(HttpClient client) { try { var payload = new MattrCrendentials { client_id = _mattrConfiguration.ClientId, client_secret = _mattrConfiguration.ClientSecret, audience = _mattrConfiguration.Audience }; var authUrl = "https://auth.mattr.global/oauth/token"; var tokenResponse = await client.PostAsJsonAsync(authUrl, payload); if (tokenResponse.StatusCode == System.Net.HttpStatusCode.OK) { var result = await tokenResponse.Content.ReadFromJsonAsync<AccessTokenItem>(); DateTime expirationTime = DateTimeOffset.FromUnixTimeSeconds(result.expires_in).DateTime; return new AccessTokenResult { AcessToken = result.access_token, ExpiresIn = expirationTime }; } _logger.LogError($"tokenResponse.IsError Status code: {tokenResponse.StatusCode}, Error: {tokenResponse.ReasonPhrase}"); throw new ApplicationException($"Status code: {tokenResponse.StatusCode}, Error: {tokenResponse.ReasonPhrase}"); } catch (Exception e) { _logger.LogError($"Exception {e}"); throw new ApplicationException($"Exception {e}"); } } private void AddToCache(string key, AccessTokenResult accessTokenItem) { var options = new DistributedCacheEntryOptions().SetSlidingExpiration(TimeSpan.FromDays(cacheExpirationInDays)); lock (_lock) { _cache.SetString(key, JsonConvert.SerializeObject(accessTokenItem), options); } } private AccessTokenResult GetFromCache(string key) { var item = _cache.GetString(key); if (item != null) { return JsonConvert.DeserializeObject<AccessTokenResult>(item); } return null; } }

Generating the API DTOs using Nswag

The MattrOpenApiClientSevice file was generated using Nswag and the Open API file provided by MATTR here. We only generated the DTOs using this and access the client then using a HttpClient instance. The Open API file used in this solution is deployed in the git repo.

Creating the OIDC credential issuer

The MattrCredentialsService is used to create an OIDC credentials issuer using the MATTR APIs. This is implemented using the CreateCredentialsAndCallback method. The created callback is returned so that it can be displayed in the UI and copied to the specific Auth0 application configuration.

private readonly IConfiguration _configuration; private readonly DriverLicenseCredentialsService _driverLicenseService; private readonly IHttpClientFactory _clientFactory; private readonly MattrTokenApiService _mattrTokenApiService; private readonly MattrConfiguration _mattrConfiguration; public MattrCredentialsService(IConfiguration configuration, DriverLicenseCredentialsService driverLicenseService, IHttpClientFactory clientFactory, IOptions<MattrConfiguration> mattrConfiguration, MattrTokenApiService mattrTokenApiService) { _configuration = configuration; _driverLicenseService = driverLicenseService; _clientFactory = clientFactory; _mattrTokenApiService = mattrTokenApiService; _mattrConfiguration = mattrConfiguration.Value; } public async Task<string> CreateCredentialsAndCallback(string name) { // create a new one var driverLicenseCredentials = await CreateMattrDidAndCredentialIssuer(); driverLicenseCredentials.Name = name; await _driverLicenseService.CreateDriverLicense(driverLicenseCredentials); var callback = $"https://{_mattrConfiguration.TenantSubdomain}/ext/oidc/v1/issuers/{driverLicenseCredentials.OidcIssuerId}/federated/callback"; return callback; }

The CreateMattrDidAndCredentialIssuer