Author: Michael G

We are pleased to announce the release of Ruby 3.2.0. Ruby 3.2 adds many features and performance improvements.

WASI based WebAssembly support

This is an initial port of WASI based WebAssembly support. This enables a CRuby binary to be available on a Web browser, a Serverless Edge environment, or other kinds of WebAssembly/WASI embedders. Currently this port passes basic and bootstrap test suites not using the Thread API.

Background

WebAssembly (Wasm) was originally introduced to run programs safely and fast in web browsers. But its objective – running programs efficiently with security on various environment – is long wanted not only for web but also by general applications.

WASI (The WebAssembly System Interface) is designed for such use cases. Though such applications need to communicate with operating systems, WebAssembly runs on a virtual machine which didn’t have a system interface. WASI standardizes it.

WebAssembly/WASI support in Ruby intends to leverage those projects. It enables Ruby developers to write applications which run on such promised platforms.

Use case

This support encourages developers to utilize CRuby in a WebAssembly environment. An example use case is TryRuby playground’s CRuby support. Now you can try original CRuby in your web browser.

Technical points

Today’s WASI and WebAssembly itself is missing some features to implement Fiber, exception, and GC because it’s still evolving, and also for security reasons. So CRuby fills the gap by using Asyncify, which is a binary transformation technique to control execution in userland.

In addition, we built a VFS on top of WASI so that we can easily pack Ruby apps into a single .wasm file. This makes distribution of Ruby apps a bit easier.

Related links

• Add WASI based WebAssembly support #5407
• An Update on WebAssembly/WASI Support in Ruby

Production-ready YJIT

• YJIT is no longer experimental
  • Has been tested on production workloads for over a year and proven to be quite stable.
• YJIT now supports both x86-64 and arm64/aarch64 CPUs on Linux, MacOS, BSD and other UNIX platforms.
  • This release brings support for Apple M1/M2, AWS Graviton, Raspberry Pi 4 and more.
• Building YJIT now requires Rust 1.58.0+. [Feature #18481]
  • In order to ensure that CRuby is built with YJIT, please install rustc >= 1.58.0
    before running the ./configure script.
  • Please reach out to the YJIT team should you run into any issues.
• The YJIT 3.2 release is faster than 3.1, and has about 1/3 as much memory overhead.
  • Overall YJIT is 41% faster (geometric mean) than the Ruby interpreter on yjit-bench.
  • Physical memory for JIT code is lazily allocated. Unlike Ruby 3.1,
    the RSS of a Ruby process is minimized because virtual memory pages
    allocated by --yjit-exec-mem-size will not be mapped to physical
    memory pages until actually utilized by JIT code.
  • Introduce Code GC that frees all code pages when the memory consumption
    by JIT code reaches --yjit-exec-mem-size.
  • RubyVM::YJIT.runtime_stats returns Code GC metrics in addition to
    existing inline_code_size and outlined_code_size keys:
    code_gc_count, live_page_count, freed_page_count, and freed_code_size.
• Most of the statistics produced by RubyVM::YJIT.runtime_stats are now available in release builds.
  • Simply run ruby with --yjit-stats to compute and dump stats (incurs some run-time overhead).
• YJIT is now optimized to take advantage of object shapes. [Feature #18776]
• Take advantage of finer-grained constant invalidation to invalidate less code when defining new constants. [Feature #18589]
• The default --yjit-exec-mem-size is changed to 64 (MiB).
• The default --yjit-call-threshold is changed to 30.

Regexp improvements against ReDoS

It is known that Regexp matching may take unexpectedly long. If your code attempts to match a possibly inefficient Regexp against an untrusted input, an attacker may exploit it for efficient Denial of Service (so-called Regular expression DoS, or ReDoS).

We have introduced two improvements that significantly mitigate ReDoS.

Improved Regexp matching algorithm

Since Ruby 3.2, Regexp’s matching algorithm has been greatly improved by using a memoization technique.

# This match takes 10 sec. in Ruby 3.1, and 0.003 sec. in Ruby 3.2

/^a*b?a*$/ =~ "a" * 50000 + "x"

The improved matching algorithm allows most Regexp matching (about 90% in our experiments) to be completed in linear time.

(For preview users: this optimization may consume memory proportional to the input length for each match. We expect no practical problems to arise because this memory allocation is usually delayed, and a normal Regexp match should consume at most 10 times as much memory as the input length. If you run out of memory when matching Regexps in a real-world application, please report it.)

The original proposal is https://bugs.ruby-lang.org/issues/19104

Regexp timeout

The optimization above cannot be applied to some kind of regular expressions, such as those including advanced features (e.g., back-references or look-around), or with a huge fixed number of repetitions. As a fallback measure, a timeout feature for Regexp matches is also introduced.

Regexp.timeout = 1.0

/^a*b?a*()1$/ =~ "a" * 50000 + "x"
#=> Regexp::TimeoutError is raised in one second

Note that Regexp.timeout is a global configuration. If you want to use different timeout settings for some special Regexps, you may want to use the timeout keyword for Regexp.new.

Regexp.timeout = 1.0

# This regexp has no timeout
long_time_re = Regexp.new('^a*b?a*()1$', timeout: Float::INFINITY)

long_time_re =~ "a" * 50000 + "x" # never interrupted

The original proposal is https://bugs.ruby-lang.org/issues/17837.

Other Notable New Features

SyntaxSuggest

• The feature of syntax_suggest (formerly dead_end) is integrated into Ruby. This helps you find the position of errors such as missing or superfluous ends, to get you back on your way faster, such as in the following example:

  Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
  
    1  class Dog
  > 2    defbark
  > 3    end
    4  end
  

  [Feature #18159]

ErrorHighlight

• Now it points at the relevant argument(s) for TypeError and ArgumentError

test.rb:2:in `+': nil can't be coerced into Integer (TypeError)

sum = ary[0] + ary[1]
               ^^^^^^

Language

• Anonymous rest and keyword rest arguments can now be passed as
  arguments, instead of just used in method parameters.
  [Feature #18351]

    def foo(*)
      bar(*)
    end
    def baz(**)
      quux(**)
    end
  

• A proc that accepts a single positional argument and keywords will
  no longer autosplat. [Bug #18633]

  proc{|a, **k| a}.call([1, 2])
  # Ruby 3.1 and before
  # => 1
  # Ruby 3.2 and after
  # => [1, 2]
  

• Constant assignment evaluation order for constants set on explicit
  objects has been made consistent with single attribute assignment
  evaluation order. With this code:

    foo::BAR = baz
  

  foo is now called before baz. Similarly, for multiple assignments
  to constants, left-to-right evaluation order is used. With this
  code:

      foo1::BAR1, foo2::BAR2 = baz1, baz2
  

  The following evaluation order is now used:

  1. foo1
  2. foo2
  3. baz1
  4. baz2

  [Bug #15928]

• The find pattern is no longer experimental.
  [Feature #18585]

• Methods taking a rest parameter (like *args) and wishing to delegate keyword
  arguments through foo(*args) must now be marked with ruby2_keywords
  (if not already the case). In other words, all methods wishing to delegate
  keyword arguments through *args must now be marked with ruby2_keywords,
  with no exception. This will make it easier to transition to other ways of
  delegation once a library can require Ruby 3+. Previously, the ruby2_keywords
  flag was kept if the receiving method took *args, but this was a bug and an
  inconsistency. A good technique to find potentially missing ruby2_keywords
  is to run the test suite, find the last method which must
  receive keyword arguments for each place where the test suite fails, and use puts nil, caller, nil there. Then check that each
  method/block on the call chain which must delegate keywords is correctly marked
  with ruby2_keywords. [Bug #18625] [Bug #16466]

    def target(**kw)
    end
  
    # Accidentally worked without ruby2_keywords in Ruby 2.7-3.1, ruby2_keywords
    # needed in 3.2+. Just like (*args, **kwargs) or (...) would be needed on
    # both #foo and #bar when migrating away from ruby2_keywords.
    ruby2_keywords def bar(*args)
      target(*args)
    end
  
    ruby2_keywords def foo(*args)
      bar(*args)
    end
  
    foo(k: 1)
  

Performance improvements

MJIT

• The MJIT compiler is re-implemented in Ruby as ruby_vm/mjit/compiler.
• MJIT compiler is executed under a forked Ruby process instead of
  doing it in a native thread called MJIT worker. [Feature #18968]
  • As a result, Microsoft Visual Studio (MSWIN) is no longer supported.
• MinGW is no longer supported. [Feature #18824]
• Rename --mjit-min-calls to --mjit-call-threshold.
• Change default --mjit-max-cache back from 10000 to 100.

PubGrub

• Bundler 2.4 now uses PubGrub resolver instead of Molinillo.

  • PubGrub is the next generation solving algorithm used by pub package manager for the Dart programming language.
  • You may get different resolution result after this change. Please report such cases to RubyGems/Bundler issues

• RubyGems still uses Molinillo resolver in Ruby 3.2. We plan to replace it with PubGrub in the future.

Other notable changes since 3.1

• Data

  • New core class to represent simple immutable value object. The class is
    similar to Struct and partially shares an implementation, but has more
    lean and strict API. [Feature #16122]

      Measure = Data.define(:amount, :unit)
      distance = Measure.new(100, 'km')            #=> #<data Measure amount=100, unit="km">
      weight = Measure.new(amount: 50, unit: 'kg') #=> #<data Measure amount=50, unit="kg">
      weight.with(amount: 40)                      #=> #<data Measure amount=40, unit="kg">
      weight.amount                                #=> 50
      weight.amount = 40                           #=> NoMethodError: undefined method `amount='
    

• Hash
  • Hash#shift now always returns nil if the hash is
    empty, instead of returning the default value or
    calling the default proc. [Bug #16908]
• MatchData
  • MatchData#byteoffset has been added. [Feature #13110]
• Module
  • Module.used_refinements has been added. [Feature #14332]
  • Module#refinements has been added. [Feature #12737]
  • Module#const_added has been added. [Feature #17881]
• Proc
  • Proc#dup returns an instance of subclass. [Bug #17545]
  • Proc#parameters now accepts lambda keyword. [Feature #15357]
• Refinement
  • Refinement#refined_class has been added. [Feature #12737]
• RubyVM::AbstractSyntaxTree
  • Add error_tolerant option for parse, parse_file and of. [Feature #19013]
    With this option
    1. SyntaxError is suppressed
    2. AST is returned for invalid input
    3. end is complemented when a parser reaches to the end of input but end is insufficient
    4. end is treated as keyword based on indent

      # Without error_tolerant option
      root = RubyVM::AbstractSyntaxTree.parse(<<~RUBY)
      def m
        a = 10
        if
      end
      RUBY
      # => <internal:ast>:33:in `parse': syntax error, unexpected `end' (SyntaxError)
    
      # With error_tolerant option
      root = RubyVM::AbstractSyntaxTree.parse(<<~RUBY, error_tolerant: true)
      def m
        a = 10
        if
      end
      RUBY
      p root # => #<RubyVM::AbstractSyntaxTree::Node:SCOPE@1:0-4:3>
    
      # `end` is treated as keyword based on indent
      root = RubyVM::AbstractSyntaxTree.parse(<<~RUBY, error_tolerant: true)
      module Z
        class Foo
          foo.
        end
    
        def bar
        end
      end
      RUBY
      p root.children[-1].children[-1].children[-1].children[-2..-1]
      # => [#<RubyVM::AbstractSyntaxTree::Node:CLASS@2:2-4:5>, #<RubyVM::AbstractSyntaxTree::Node:DEFN@6:2-7:5>]
    

  • Add keep_tokens option for parse, parse_file and of. [Feature #19070]

      root = RubyVM::AbstractSyntaxTree.parse("x = 1 + 2", keep_tokens: true)
      root.tokens # => [[0, :tIDENTIFIER, "x", [1, 0, 1, 1]], [1, :tSP, " ", [1, 1, 1, 2]], ...]
      root.tokens.map{_1[2]}.join # => "x = 1 + 2"
    

• Set
  • Set is now available as a builtin class without the need for require "set". [Feature #16989]
    It is currently autoloaded via the Set constant or a call to Enumerable#to_set.
• String
  • String#byteindex and String#byterindex have been added. [Feature #13110]
  • Update Unicode to Version 15.0.0 and Emoji Version 15.0. [Feature #18639]
    (also applies to Regexp)
  • String#bytesplice has been added. [Feature #18598]
• Struct

  • A Struct class can also be initialized with keyword arguments
    without keyword_init: true on Struct.new [Feature #16806]

      Post = Struct.new(:id, :name)
      Post.new(1, "hello") #=> #<struct Post id=1, name="hello">
      # From Ruby 3.2, the following code also works without keyword_init: true.
      Post.new(id: 1, name: "hello") #=> #<struct Post id=1, name="hello">
    

Compatibility issues

Note: Excluding feature bug fixes.

Removed constants

The following deprecated constants are removed.

• Fixnum and Bignum [Feature #12005]
• Random::DEFAULT [Feature #17351]
• Struct::Group
• Struct::Passwd

Removed methods

The following deprecated methods are removed.

• Dir.exists? [Feature #17391]
• File.exists? [Feature #17391]
• Kernel#=~ [Feature #15231]
• Kernel#taint, Kernel#untaint, Kernel#tainted?
  [Feature #16131]
• Kernel#trust, Kernel#untrust, Kernel#untrusted?
  [Feature #16131]

Stdlib compatibility issues

No longer bundle 3rd party sources

• We no longer bundle 3rd party sources like libyaml, libffi.

  • libyaml source has been removed from psych. You may need to install libyaml-dev with Ubuntu/Debian platform. The package name is different for each platform.

  • Bundled libffi source is also removed from fiddle

• Psych and fiddle supported static builds with specific versions of libyaml and libffi sources. You can build psych with libyaml-0.2.5 like this:

    $ ./configure --with-libyaml-source-dir=/path/to/libyaml-0.2.5
  

  And you can build fiddle with libffi-3.4.4 like this:

    $ ./configure --with-libffi-source-dir=/path/to/libffi-3.4.4
  

  [Feature #18571]

C API updates

Updated C APIs

The following APIs are updated.

• PRNG update
  • rb_random_interface_t updated and versioned.
    Extension libraries which use this interface and built for older versions.
    Also init_int32 function needs to be defined.

Removed C APIs

The following deprecated APIs are removed.

• rb_cData variable.
• “taintedness” and “trustedness” functions. [Feature #16131]

Standard library updates

• Bundler

  • Add –ext=rust support to bundle gem for creating simple gems with Rust extensions.
    [GH-rubygems-6149]
  • Make cloning git repos faster [GH-rubygems-4475]

• RubyGems

  • Add mswin support for cargo builder. [GH-rubygems-6167]

• ERB

  • ERB::Util.html_escape is made faster than CGI.escapeHTML.
    • It no longer allocates a String object when no character needs to be escaped.
    • It skips calling #to_s method when an argument is already a String.
    • ERB::Escape.html_escape is added as an alias to ERB::Util.html_escape,
      which has not been monkey-patched by Rails.

• IRB

  • debug.gem integration commands have been added: debug, break, catch,
    next, delete, step, continue, finish, backtrace, info
    • They work even if you don’t have gem "debug" in your Gemfile.
    • See also: What’s new in Ruby 3.2’s IRB?
  • More Pry-like commands and features have been added.
    • edit and show_cmds (like Pry’s help) are added.
    • ls takes -g or -G option to filter out outputs.
    • show_source is aliased from $ and accepts unquoted inputs.
    • whereami is aliased from @.

• The following default gems are updated.

  • RubyGems 3.4.1
  • abbrev 0.1.1
  • benchmark 0.2.1
  • bigdecimal 3.1.3
  • bundler 2.4.1
  • cgi 0.3.6
  • csv 3.2.6
  • date 3.3.3
  • delegate 0.3.0
  • did_you_mean 1.6.3
  • digest 3.1.1
  • drb 2.1.1
  • english 0.7.2
  • erb 4.0.2
  • error_highlight 0.5.1
  • etc 1.4.2
  • fcntl 1.0.2
  • fiddle 1.1.1
  • fileutils 1.7.0
  • forwardable 1.3.3
  • getoptlong 0.2.0
  • io-console 0.6.0
  • io-nonblock 0.2.0
  • io-wait 0.3.0
  • ipaddr 1.2.5
  • irb 1.6.2
  • json 2.6.3
  • logger 1.5.3
  • mutex_m 0.1.2
  • net-http 0.3.2
  • net-protocol 0.2.1
  • nkf 0.1.2
  • open-uri 0.3.0
  • open3 0.1.2
  • openssl 3.1.0
  • optparse 0.3.1
  • ostruct 0.5.5
  • pathname 0.2.1
  • pp 0.4.0
  • pstore 0.1.2
  • psych 5.0.1
  • racc 1.6.2
  • rdoc 6.5.0
  • readline-ext 0.1.5
  • reline 0.3.2
  • resolv 0.2.2
  • resolv-replace 0.1.1
  • securerandom 0.2.2
  • set 1.0.3
  • stringio 3.0.4
  • strscan 3.0.5
  • syntax_suggest 1.0.2
  • syslog 0.1.1
  • tempfile 0.1.3
  • time 0.2.1
  • timeout 0.3.1
  • tmpdir 0.1.3
  • tsort 0.1.1
  • un 0.2.1
  • uri 0.12.0
  • weakref 0.1.2
  • win32ole 1.8.9
  • yaml 0.2.1
  • zlib 3.0.0

• The following bundled gems are updated.

  • minitest 5.16.3
  • power_assert 2.0.3
  • test-unit 3.5.7
  • net-ftp 0.2.0
  • net-imap 0.3.3
  • net-pop 0.1.2
  • net-smtp 0.3.3
  • rbs 2.8.2
  • typeprof 0.21.3
  • debug 1.7.1

See GitHub releases like GitHub Releases of logger or changelog for details of the default gems or bundled gems.

See NEWS
or commit logs
for more details.

With those changes, 3048 files changed, 218253 insertions(+), 131067 deletions(-)
since Ruby 3.1.0!

Merry Christmas, Happy Holidays, and enjoy programming with Ruby 3.2!

Download

• https://cache.ruby-lang.org/pub/ruby/3.2/ruby-3.2.0.tar.gz

  SIZE: 20440715
  SHA1: fb4ab2ceba8bf6a5b9bc7bf7cac945cc94f94c2b
  SHA256: daaa78e1360b2783f98deeceb677ad900f3a36c0ffa6e2b6b19090be77abc272
  SHA512: 94203051d20475b95a66660016721a0457d7ea57656a9f16cdd4264d8aa6c4cd8ea2fab659082611bfbd7b00ebbcf0391e883e2ebf384e4fab91869e0a877d35
  

• https://cache.ruby-lang.org/pub/ruby/3.2/ruby-3.2.0.tar.xz

  SIZE: 15058364
  SHA1: bcdae07183d66fd902cb7bf995545a472d2fefea
  SHA256: d2f4577306e6dd932259693233141e5c3ec13622c95b75996541b8d5b68b28b4
  SHA512: 733ecc6709470ee16916deeece9af1c76220ae95d17b2681116aff7f381d99bc3124b1b11b1c2336b2b29e468e91b90f158d5ae5fca810c6cf32a0b6234ae08e
  

• https://cache.ruby-lang.org/pub/ruby/3.2/ruby-3.2.0.zip

  SIZE: 24583271
  SHA1: 581ec7b9289c2a85abf4f41c93993ecaa5cf43a5
  SHA256: cca9ddbc958431ff77f61948cb67afa569f01f99c9389d2bbedfa92986c9ef09
  SHA512: b7d2753825cc0667e8bb391fc7ec59a53c3db5fa314e38eee74b6511890b585ac7515baa2ddac09e2c6b6c42b9221c82e040af5b39c73e980fbd3b1bc622c99d
  

What is Ruby

Ruby was first developed by Matz (Yukihiro Matsumoto) in 1993,
and is now developed as Open Source. It runs on multiple platforms
and is used all over the world especially for web development.

Posted by naruse on 25 Dec 2022

I use GnuPG to compute cryptographic signatures for my emails, git commits/tags, and software release artifacts (tarballs). Part of GnuPG is gpg-agent which talks to OpenSSH, which I login to remote servers and to clone git repositories. I dislike storing cryptographic keys on general-purpose machines, and have used hardware-backed OpenPGP keys since around 2006 when I got a FSFE Fellowship Card. GnuPG via gpg-agent handles this well, and the private key never leaves the hardware. These ZeitControl cards were (to my knowledge) proprietary hardware running some non-free operating system and OpenPGP implementation. By late 2012 the YubiKey NEO supported OpenPGP, and while the hardware and operating system on it was not free, at least it ran a free software OpenPGP implementation and eventually I setup my primary RSA key on it. This worked well for a couple of years, and when I in 2019 wished to migrate to a new key, the FST-01G device with open hardware running free software that supported Ed25519 had become available. I created a key and have been using the FST-01G on my main laptop since then. This little device has been working, the signature counter on it is around 14501 which means around 10 signatures/day since then!

Currently I am in the process of migrating towards a new laptop, and moving the FST-01G device between them is cumbersome, especially if I want to use both laptops in parallel. That’s why I need to setup a new hardware device to hold my OpenPGP key, which can go with my new laptop. This is a good time to re-visit earlier options again. I quickly decided that I did not want to create a new key, only to import my current one to keep everything working. My requirements on the device to chose hasn’t changed since 2019, see my summary at the end of the earlier blog post. Unfortunately the FST-01G is not available on the market, and the newer FST-01SZ has been out of stock for quite a while. While Tillitis looks promising (and I have one to play with), it does not support OpenPGP (yet). What to do? Fortunately, I found some FST-01SZ device in my drawer, and decided to use it pending a more satisfactory answer. Hopefully once I get around to generate a new OpenPGP key in a year or so, I will do a better survey of options that are available on the market then. What are your (freedom-respecting) OpenPGP hardware recommendations?

Similar to setting up the FST-01G, the FST-01SZ needs to be setup before use. I’m doing the following from Trisquel 11 but any GNU/Linux system would work. When the device is inserted at first time, some kernel messages are shown (see /var/log/syslog or use the dmesg command):

usb 3-3: new full-speed USB device number 39 using xhci_hcd
usb 3-3: New USB device found, idVendor=234b, idProduct=0004, bcdDevice= 2.00
usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-3: Product: Fraucheky
usb 3-3: Manufacturer: Free Software Initiative of Japan
usb 3-3: SerialNumber: FSIJ-0.0
usb-storage 3-3:1.0: USB Mass Storage device detected
scsi host1: usb-storage 3-3:1.0
scsi 1:0:0:0: Direct-Access     FSIJ     Fraucheky        1.0  PQ: 0 ANSI: 0
sd 1:0:0:0: Attached scsi generic sg2 type 0
sd 1:0:0:0: [sdc] 128 512-byte logical blocks: (65.5 kB/64.0 KiB)
sd 1:0:0:0: [sdc] Write Protect is off
sd 1:0:0:0: [sdc] Mode Sense: 03 00 00 00
sd 1:0:0:0: [sdc] No Caching mode page found
sd 1:0:0:0: [sdc] Assuming drive cache: write through
 sdc:
sd 1:0:0:0: [sdc] Attached SCSI removable disk

Interestingly, the NeuG software installed on the device I got appears to be version 1.0.9:

jas@kaka:~$ head /media/jas/Fraucheky/README
NeuG - a true random number generator implementation
						  Version 1.0.9
						     2018-11-20
					           Niibe Yutaka
			      Free Software Initiative of Japan
What's NeuG?
============
jas@kaka:~$ 

I could not find version 1.0.9 published anywhere, but the device came with a SD-card that contain a copy of the source, so I uploaded it until a more canonical place is located. Putting the device in the serial mode can be done using a sudo eject /dev/sdc command which results in the following syslog output.

usb 3-3: reset full-speed USB device number 39 using xhci_hcd
usb 3-3: device firmware changed
usb 3-3: USB disconnect, device number 39
sdc: detected capacity change from 128 to 0
usb 3-3: new full-speed USB device number 40 using xhci_hcd
usb 3-3: New USB device found, idVendor=234b, idProduct=0001, bcdDevice= 2.00
usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-3: Product: NeuG True RNG
usb 3-3: Manufacturer: Free Software Initiative of Japan
usb 3-3: SerialNumber: FSIJ-1.0.9-42315277
cdc_acm 3-3:1.0: ttyACM0: USB ACM device

Now download Gnuk, verify its integrity and build it. You may need some additional packages installed, try apt-get install gcc-arm-none-eabi openocd python3-usb. As you can see, I’m using the stable 1.2 branch of Gnuk, currently on version 1.2.20. The ./configure parameters deserve some explanation. The kdf_do=required sets up the device to require KDF usage. The --enable-factory-reset allows me to use the command factory-reset (with admin PIN) inside gpg --card-edit to completely wipe the card. Some may consider that too dangerous, but my view is that if someone has your admin PIN it is game over anyway. The --vidpid=234b:0000 is specifies the USB VID/PID to use, and --target=FST_01SZ is critical to set the platform (you’ll may brick the device if you pick the wrong --target setting).

jas@kaka:~/src$ rm -rf gnuk neug
jas@kaka:~/src$ git clone https://gitlab.com/jas/neug.git
Cloning into 'neug'...
remote: Enumerating objects: 2034, done.
remote: Counting objects: 100% (2034/2034), done.
remote: Compressing objects: 100% (603/603), done.
remote: Total 2034 (delta 1405), reused 2013 (delta 1405), pack-reused 0
Receiving objects: 100% (2034/2034), 910.34 KiB | 3.50 MiB/s, done.
Resolving deltas: 100% (1405/1405), done.
jas@kaka:~/src$ git clone https://salsa.debian.org/gnuk-team/gnuk/gnuk.git
Cloning into 'gnuk'...
remote: Enumerating objects: 13765, done.
remote: Counting objects: 100% (959/959), done.
remote: Compressing objects: 100% (337/337), done.
remote: Total 13765 (delta 629), reused 907 (delta 599), pack-reused 12806
Receiving objects: 100% (13765/13765), 12.59 MiB | 3.05 MiB/s, done.
Resolving deltas: 100% (10077/10077), done.
jas@kaka:~/src$ cd neug
jas@kaka:~/src/neug$ git describe 
release/1.0.9
jas@kaka:~/src/neug$ git tag -v `git describe`
object 5d51022a97a5b7358d0ea62bbbc00628c6cec06a
type commit
tag release/1.0.9
tagger NIIBE Yutaka <gniibe@fsij.org> 1542701768 +0900

Version 1.0.9.
gpg: Signature made Tue Nov 20 09:16:08 2018 CET
gpg:                using EDDSA key 249CB3771750745D5CDD323CE267B052364F028D
gpg:                issuer "gniibe@fsij.org"
gpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]
gpg:                 aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 249C B377 1750 745D 5CDD  323C E267 B052 364F 028D
jas@kaka:~/src/neug$ cd ../gnuk/
jas@kaka:~/src/gnuk$ git checkout STABLE-BRANCH-1-2 
Branch 'STABLE-BRANCH-1-2' set up to track remote branch 'STABLE-BRANCH-1-2' from 'origin'.
Switched to a new branch 'STABLE-BRANCH-1-2'
jas@kaka:~/src/gnuk$ git describe
release/1.2.20
jas@kaka:~/src/gnuk$ git tag -v `git describe`
object 9d3c08bd2beb73ce942b016d4328f0a596096c02
type commit
tag release/1.2.20
tagger NIIBE Yutaka <gniibe@fsij.org> 1650594032 +0900

Gnuk: Version 1.2.20
gpg: Signature made Fri Apr 22 04:20:32 2022 CEST
gpg:                using EDDSA key 249CB3771750745D5CDD323CE267B052364F028D
gpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]
gpg:                 aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 249C B377 1750 745D 5CDD  323C E267 B052 364F 028D
jas@kaka:~/src/gnuk/src$ git submodule update --init
Submodule 'chopstx' (https://salsa.debian.org/gnuk-team/chopstx/chopstx.git) registered for path '../chopstx'
Cloning into '/home/jas/src/gnuk/chopstx'...
Submodule path '../chopstx': checked out 'e12a7e0bb3f004c7bca41cfdb24c8b66daf3db89'
jas@kaka:~/src/gnuk$ cd chopstx
jas@kaka:~/src/gnuk/chopstx$ git describe
release/1.21
jas@kaka:~/src/gnuk/chopstx$ git tag -v `git describe`
object e12a7e0bb3f004c7bca41cfdb24c8b66daf3db89
type commit
tag release/1.21
tagger NIIBE Yutaka <gniibe@fsij.org> 1650593697 +0900

Chopstx: Version 1.21
gpg: Signature made Fri Apr 22 04:14:57 2022 CEST
gpg:                using EDDSA key 249CB3771750745D5CDD323CE267B052364F028D
gpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]
gpg:                 aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 249C B377 1750 745D 5CDD  323C E267 B052 364F 028D
jas@kaka:~/src/gnuk/chopstx$ cd ../src
jas@kaka:~/src/gnuk/src$ kdf_do=required ./configure --enable-factory-reset --vidpid=234b:0000 --target=FST_01SZ
Header file is: board-fst-01sz.h
Debug option disabled
Configured for bare system (no-DFU)
PIN pad option disabled
CERT.3 Data Object is NOT supported
Card insert/removal by HID device is NOT supported
Life cycle management is supported
Acknowledge button is supported
KDF DO is required before key import/generation
jas@kaka:~/src/gnuk/src$ make | less
jas@kaka:~/src/gnuk/src$ cd ../regnual/
jas@kaka:~/src/gnuk/regnual$ make | less
jas@kaka:~/src/gnuk/regnual$ cd ../../
jas@kaka:~/src$ sudo python3 neug/tool/neug_upgrade.py -f gnuk/regnual/regnual.bin gnuk/src/build/gnuk.bin
gnuk/regnual/regnual.bin: 4608
gnuk/src/build/gnuk.bin: 109568
CRC32: b93ca829

Device: 
Configuration: 1
Interface: 1
20000e00:20005000
Downloading flash upgrade program...
start 20000e00
end   20002000
# 20002000: 32 : 4
Run flash upgrade program...
Wait 1 second...
Wait 1 second...
Device: 
08001000:08020000
Downloading the program
start 08001000
end   0801ac00
jas@kaka:~/src$ 

The kernel log will contain the following, and the card is ready to use as an OpenPGP card. You may unplug it and re-insert it as you wish.

usb 3-3: reset full-speed USB device number 41 using xhci_hcd
usb 3-3: device firmware changed
usb 3-3: USB disconnect, device number 41
usb 3-3: new full-speed USB device number 42 using xhci_hcd
usb 3-3: New USB device found, idVendor=234b, idProduct=0000, bcdDevice= 2.00
usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-3: Product: Gnuk Token
usb 3-3: Manufacturer: Free Software Initiative of Japan
usb 3-3: SerialNumber: FSIJ-1.2.20-42315277

Setting up the card is the next step, and there are many tutorials around for this, eventually I settled with the following sequence. Let’s start with setting the admin PIN. First make sure that pcscd nor scdaemon is running, which is good hygien since those processes cache some information and with a stale connection this easily leads to confusion. Cache invalidation… sigh.

jas@kaka:~$ gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye
jas@kaka:~$ ps auxww|grep -e pcsc -e scd
jas        30221  0.0  0.0   3468  1692 pts/3    R+   11:49   0:00 grep --color=auto -e pcsc -e scd
jas@kaka:~$ gpg --card-edit

Reader ...........: 234B:0000:FSIJ-1.2.20-42315277:0
Application ID ...: D276000124010200FFFE423152770000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 42315277
Name of cardholder: [not set]
Language prefs ...: [not set]
Salutation .......: 
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> admin
Admin commands are allowed

gpg/card> kdf-setup

gpg/card> passwd
gpg: OpenPGP card no. D276000124010200FFFE423152770000 detected

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 3
PIN changed.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 

Now it would be natural to setup the PIN and reset code. However the Gnuk software is configured to not allow this until the keys are imported. You would get the following somewhat cryptical error messages if you try. This took me a while to understand, since this is device-specific, and some other OpenPGP implementations allows you to configure a PIN and reset code before key import.

Your selection? 4
Error setting the Reset Code: Card error

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 1
Error changing the PIN: Conditions of use not satisfied

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? q

Continue to configure the card and make it ready for key import. Some settings deserve comments. The lang field may be used to setup the language, but I have rarely seen it use, and I set it to ‘sv‘ (Swedish) mostly to be able to experiment if any software adhears to it. The URL is important to point to somewhere where your public key is stored, the fetch command of gpg --card-edit downloads it and sets up GnuPG with it when you are on a clean new laptop. The forcesig command changes the default so that a PIN code is not required for every digital signature operation, remember that I averaged 10 signatures per day for the past 2-3 years? Think of the wasted energy typing those PIN codes every time! Changing the cryptographic key type is required when I import 25519-based keys.

gpg/card> name
Cardholder's surname: Josefsson
Cardholder's given name: Simon

gpg/card> lang
Language preferences: sv

gpg/card> sex
Salutation (M = Mr., F = Ms., or space): m

gpg/card> login
Login data (account name): jas

gpg/card> url
URL to retrieve public key: https://josefsson.org/key-20190320.txt

gpg/card> forcesig

gpg/card> key-attr
Changing card key attribute for: Signature key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 2
Please select which elliptic curve you want:
   (1) Curve 25519
   (4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
Note: There is no guarantee that the card supports the requested size.
      If the key generation does not succeed, please check the
      documentation of your card to see what sizes are allowed.
Changing card key attribute for: Encryption key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 2
Please select which elliptic curve you want:
   (1) Curve 25519
   (4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: cv25519
Changing card key attribute for: Authentication key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 2
Please select which elliptic curve you want:
   (1) Curve 25519
   (4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519

gpg/card> 

Reader ...........: 234B:0000:FSIJ-1.2.20-42315277:0
Application ID ...: D276000124010200FFFE423152770000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 42315277
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: on
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> 

The device is now ready for key import! Bring out your offline laptop and boot it and use the keytocard command on the subkeys to import them. This assumes you saved a copy of the GnuPG home directory after generating the master and subkeys before, which I did in my own previous tutorial when I generated the keys. This may be a bit unusual, and there are simpler ways to do this (e.g., import a copy of the secret keys into a fresh GnuPG home directory).

$ cp -a gnupghome-backup-mastersubkeys gnupghome-import-fst01sz-42315277-2022-12-24
$ ps auxww|grep -e pcsc -e scd
$ gpg --homedir $PWD/gnupghome-import-fst01sz-42315277-2022-12-24 --edit-key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
...
Secret key is available.

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 1

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb* cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb* cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 1

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 2

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb* ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb* ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 2

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 3

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb* ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb* ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> quit
Save changes? (y/N) y
$ 

Now insert it into your daily laptop and have GnuPG and learn about the new private keys and forget about any earlier locally available card bindings — this usually manifests itself by GnuPG asking you to insert a OpenPGP card with another serial number. Earlier I did rm -rf ~/.gnupg/private-keys-v1.d/ but the scd serialno followed by learn --force is nicer. I also sets up trust setting for my own key.

jas@kaka:~$ gpg-connect-agent "scd serialno" "learn --force" /bye
...
jas@kaka:~$ echo "B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE:6:" | gpg --import-ownertrust
jas@kaka:~$ gpg --card-status
Reader ...........: 234B:0000:FSIJ-1.2.20-42315277:0
Application ID ...: D276000124010200FFFE423152770000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 42315277
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 5 5 5
Signature counter : 3
KDF setting ......: on
Signature key ....: A3CC 9C87 0B9D 310A BAD4  CF2F 5172 2B08 FE47 45A2
      created ....: 2019-03-20 23:40:49
Encryption key....: A9EC 8F4D 7F1E 50ED 3DEF  49A9 0292 3D7E E76E BD60
      created ....: 2019-03-20 23:40:26
Authentication key: CA7E 3716 4342 DF31 33DF  3497 8026 0EE8 A9B9 2B2B
      created ....: 2019-03-20 23:40:37
General key info..: sub  ed25519/51722B08FE4745A2 2019-03-20 Simon Josefsson <simon@josefsson.org>
sec#  ed25519/D73CF638C53C06BE  created: 2019-03-20  expires: 2023-09-19
ssb>  ed25519/80260EE8A9B92B2B  created: 2019-03-20  expires: 2023-09-19
                                card-no: FFFE 42315277
ssb>  ed25519/51722B08FE4745A2  created: 2019-03-20  expires: 2023-09-19
                                card-no: FFFE 42315277
ssb>  cv25519/02923D7EE76EBD60  created: 2019-03-20  expires: 2023-09-19
                                card-no: FFFE 42315277
jas@kaka:~$ 

Verify that you can digitally sign and authenticate using the key and you are done!

jas@kaka:~$ echo foo|gpg -a --sign|gpg --verify
gpg: Signature made Sat Dec 24 13:49:59 2022 CET
gpg:                using EDDSA key A3CC9C870B9D310ABAD4CF2F51722B08FE4745A2
gpg: Good signature from "Simon Josefsson <simon@josefsson.org>" [ultimate]
jas@kaka:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILzCFcHHrKzVSPDDarZPYqn89H5TPaxwcORgRg+4DagE cardno:FFFE42315277
jas@kaka:~$ 

So time to relax and celebrate christmas? Hold on… not so fast! Astute readers will have noticed that the output said ‘PIN retry counter: 5 5 5‘. That’s not the default PIN retry counter for Gnuk! How did that happen? Indeed, good catch and great question, my dear reader. I wanted to include how you can modify the Gnuk source code, re-build it and re-flash the Gnuk as well. This method is different than flashing Gnuk onto a device that is running NeuG so the commands I used to flash the firmware in the start of this blog post no longer works in a device running Gnuk. Fortunately modern Gnuk supports updating firmware by specifying the Admin PIN code only, and provides a simple script to achieve this as well. The PIN retry counter setting is hard coded in the openpgp-do.c file, and we run a a perl command to modify the file, rebuild Gnuk and upgrade the FST-01SZ. This of course wipes all your settings, so you will have the opportunity to practice all the commands earlier in this post once again!

jas@kaka:~/src/gnuk/src$ perl -pi -e 's/PASSWORD_ERRORS_MAX 3/PASSWORD_ERRORS_MAX 5/' openpgp-do.c
jas@kaka:~/src/gnuk/src$ make | less
jas@kaka:~/src/gnuk/src$ cd ../tool/
jas@kaka:~/src/gnuk/tool$ ./upgrade_by_passwd.py 
Admin password: 
Device: 
Configuration: 1
Interface: 0
../regnual/regnual.bin: 4608
../src/build/gnuk.bin: 110592
CRC32: b93ca829

Device: 
Configuration: 1
Interface: 0
20002a00:20005000
Downloading flash upgrade program...
start 20002a00
end   20003c00
Run flash upgrade program...
Waiting for device to appear:
  Wait 1 second...
  Wait 1 second...
Device: 
08001000:08020000
Downloading the program
start 08001000
end   0801b000
Protecting device
Finish flashing
Resetting device
Update procedure finished
jas@kaka:~/src/gnuk/tool$

Now finally, I wish you all a Merry Christmas and Happy Hacking!