It’s time to transform and improve the Layout Builder UI experience. What if we could rewrite it using React? Check out our video pitch (made using React), which received the highest average rank in the Pitch-burgh contest at DrupalCon!

by
lee.rowlands
/ 6 June 2023

At PreviousNext, we enjoy getting curious, tackling challenges and finding innovative solutions for our clients and the wider Drupal community. This constant drive for the future is why we’re such prolific contributors to the Drupal project. 

What if we used our curiosity and expertise to explore the concept of a decoupled Layout Builder?

 

The Pitch

Simply put, we believe creating a better experience for the Layout Builder user interface is possible. Getting there would mean leveraging a modern Javascript framework, like React, that communicates with Drupal over JSON.

We would plan to design an API that describes the future state of how a Decoupled Layout Builder could work, dramatically enhancing the content editor experience and bringing Drupal into the modern era of web UX. It would also strengthen Drupal’s reputation with marketers.

 

The case for an improved Layout Builder user experience

What if Layout Builder could move forward? It’s currently built with Drupal technologies that haven’t significantly progressed in ten years.

Page edits use Drupal’s AJAX API, which requires a round trip back to the server to perform UI updates. The server updates state on the server side and then returns a series of Javascript commands to update the page state.

This round trip is inconsistent with the instant experience users expect from the modern web. 

 

Our solution

Applications that leverage modern Javascript frameworks perform optimistic updates. They update the UI immediately and then update state on the server in the background.

We want layout retrieval and updates to happen in real time for more dynamic editing. To achieve this, we would rebuild the Layout Builder UI using React and identify the pain points.

This would be a significant jump, similar to the change to the block-based editor in WordPress.

Rather than starting from scratch, we could leverage a lot of prior art from the WordPress community. However, we would also bring our strengths into play, retaining our uniquely Drupal focus on structured data instead of serialising to HTML. 

 

Proof of concept

React is the best and most obvious option for the proof of concept. It was selected as the framework of choice for Drupal core development and has a large ecosystem. It was also successful in helping set WordPress up for the future. 

What if it’s Drupal’s turn next?

To retain structured data, it would be necessary to provide React versions of our existing formatters and widgets and a way for developers to create their own.

This approach would require a Javascript way of declaring layouts. Again, we could take a lot of this from existing layout plugins but would require a React component for the HTML representation.

We would also need to incorporate a Javascript way of declaring blocks. Much could be derived from our existing structured content modelling. However, assuming there would be a build step where this data is used for scaffolding Javascript code is reasonable.

JSON:API could be leveraged where possible, but we envisage needing to make new JSON endpoints for data that doesn’t map to entities.

Once we had achieved a non-twig way of rendering widgets, formatters, blocks and layouts, we could keep the layout state in the browser, mutate it immediately and persist to the backend in the background.

 

Is a decoupled Layout Builder feasible?

There are a number of hard problems we’ll need to solve here, such as how we allow modules to ship Javascript that relies on bundling, how we ensure there’s only one version of React loaded into the page, and how we allow modules to rely on other packages from npm. Solving those will be a big part of this work, and even if a React-powered Layout Builder doesn’t make its way from experimental to stable, the lessons learned in the process could solve some long-standing hard problems for Drupal in the front end space.

 

React in video

We decided it would be a fitting way to demonstrate how powerful React is by using it to make our Pitch-burgh video submission. 

We received the highest average rank when the video was shown during the Driesnote address at DrupalCon Pittsburgh, and look forward to the next steps for this exciting concept. 

Huge shout out to the team who worked on creating the pitch video especially the wizard of words Fiona, frontend Guru Jack and the excellent QA and GIF selection from Kim and Tina.

Enjoy!

In this episode, we discuss the making of Avalon using Dragon Ruby with James Stocks: https://www.rubberduckdevshow.com/episodes/87-the-making-of-avalon-in-dragon-ruby-with-james-stocks/

Version 0.89
of
Game of Trees
has been released (and the port
updated):

* got 0.89; 2023-06-05
  see git repository history for per-change authorship information
- gotd: return early after disconnect on auth event error instead of crashing
- make 'got patch' display statistics about files with conflicts and rejects
- make 'got diff' not treat rn line endings as special
- fix test failures in test_blame_lines_shifted_skip on certain times of day
- show reference labels next to commit messages in tog log view
- some gotwebd refactoring related to handling of file descriptors
- gotwebd: lower log priority of unexpected disconnections
- gotwebd: avoid needless double fseek()
- fix the size of gotwebd's tempfiles array; exposed by errors from ftruncate()
- simplify ancestry checks in checkout, update, rebase, and merge commands
- make gitwrapper not fail if programs it wants to run do not exist on disk
- stop showing backup references in the tog log and diff views
- consistently use ten Xs in mkstemp(3) templates
- only delete empty directories which appear in arguments to 'got rm'
- simplify parsing of host names and IP addresses in gotwebd's parse.y
- make 'got merge' refuse to run if a merge is in progress
- make 'got merge -c' fail even if new changes only affect unrelated paths

Guix is a handy tool for developers; guix shell,
in particular, gives a standalone development environment for your
package, no matter what language(s) it’s written in. To benefit from
it, you have to initially write a package definition and have it either
in Guix proper, in a channel, or directly upstream as a guix.scm file.
This last option is appealing: all developers have to do to get set up
is clone the project’s repository and run guix shell, with no
arguments—we looked at the rationale for guix shell in an earlier
article.

Development needs go beyond development environments though. How can
developers perform continuous integration of their code in Guix build
environments? How can they deliver their code straight to adventurous
users? This post describes a set of files developers can add
to their repository to set up Guix-based development
environments, continuous integration, and continuous delivery—all at
once.

Getting started

How do we go about “Guixifying” a repository? The first step, as we’ve
seen, will be to add a guix.scm at the root of the repository in
question. We’ll take Guile
as an example in this post: it’s written in Scheme (mostly) and C, and
has a number of dependencies—a C compilation tool chain, C libraries,
Autoconf and its friends, LaTeX, and so on. The resulting guix.scm
looks like the usual package
definition,
just without the define-public bit:

;; The ‘guix.scm’ file for Guile, for use by ‘guix shell’.

(use-modules (guix)
             (guix build-system gnu)
             ((guix licenses) #:prefix license:)
             (gnu packages autotools)
             (gnu packages base)
             (gnu packages bash)
             (gnu packages bdw-gc)
             (gnu packages compression)
             (gnu packages flex)
             (gnu packages gdb)
             (gnu packages gettext)
             (gnu packages gperf)
             (gnu packages libffi)
             (gnu packages libunistring)
             (gnu packages linux)
             (gnu packages pkg-config)
             (gnu packages readline)
             (gnu packages tex)
             (gnu packages texinfo)
             (gnu packages version-control))

(package
  (name "guile")
  (version "3.0.99-git")                          ;funky version number
  (source #f)                                     ;no source
  (build-system gnu-build-system)
  (native-inputs
   (append (list autoconf
                 automake
                 libtool
                 gnu-gettext
                 flex
                 texinfo
                 texlive-base                 ;for "make pdf"
                 texlive-epsf
                 gperf
                 git
                 gdb
                 strace
                 readline
                 lzip
                 pkg-config)

           ;; When cross-compiling, a native version of Guile itself is
           ;; needed.
           (if (%current-target-system)
               (list this-package)
               '())))
  (inputs
   (list libffi bash-minimal))
  (propagated-inputs
   (list libunistring libgc))

  (native-search-paths
   (list (search-path-specification
          (variable "GUILE_LOAD_PATH")
          (files '("share/guile/site/3.0")))
         (search-path-specification
          (variable "GUILE_LOAD_COMPILED_PATH")
          (files '("lib/guile/3.0/site-ccache")))))
  (synopsis "Scheme implementation intended especially for extensions")
  (description
   "Guile is the GNU Ubiquitous Intelligent Language for Extensions,
and it's actually a full-blown Scheme implementation!")
  (home-page "https://www.gnu.org/software/guile/")
  (license license:lgpl3+))

Quite a bit of boilerplate, but now someone who’d like to hack on Guile
just needs to run:

guix shell

That gives them a shell containing all the dependencies of Guile: those
listed above, but also implicit dependencies such as the GCC tool
chain, GNU Make, sed, grep, and so on. The chef’s recommendation:

guix shell --container --link-profile

That gives a shell in an isolated container, and all the dependencies
show up in $HOME/.guix-profile, which plays well with caches such as
config.cache
and absolute file names recorded in generated Makefiles and the likes.
The fact that the shell runs in a container brings peace of mind:
nothing but the current directory and Guile’s dependencies is visible
inside the container; nothing from the system can possibly interfere
with your development.

Level 1: Building with Guix

Now that we have a package definition, why not also take advantage of it
so we can build Guile with Guix? We had left the source field empty,
because guix shell above only cares about the inputs of our
package—so it can set up the development environment—not about the
package itself.

To build the package with Guix, we’ll need to fill out the source
field, along these lines:

(use-modules (guix)
             (guix git-download)  ;for ‘git-predicate’
             …)

(define vcs-file?
  ;; Return true if the given file is under version control.
  (or (git-predicate (current-source-directory))
      (const #t)))                                ;not in a Git checkout

(package
  (name "guile")
  (version "3.0.99-git")                          ;funky version number
  (source (local-file "." "guile-checkout"
                      #:recursive? #t
                      #:select? vcs-file?))
  …)

Here’s what we changed:

1. We added (guix git-download) to our set of imported modules, so we
   can use its git-predicate procedure.
2. We defined vcs-file? as a procedure that returns true when passed
   a file that is under version control. For good measure, we add a
   fallback case for when we’re not in a Git checkout: always return
   true.
3. We set source to a
   local-file—a
   recursive copy of the current directory ("."), limited to files
   under version control (the #:select? bit).

From there on, our guix.scm file serves a second purpose: it lets us
build the software with Guix. The whole point of building with Guix is
that it’s a “clean” build—you can be sure nothing from your working tree
or system interferes with the build result—and it lets you test a
variety of things. First, you can do a plain native build:

guix build -f guix.scm

But you can also build for another system (possibly after setting up
offloading
or transparent
emulation):

guix build -f guix.scm -s aarch64-linux -s riscv64-linux

… or cross-compile:

guix build -f guix.scm --target=x86_64-w64-mingw32

You can also use package transformation
options
to test package variants:

# What if we built with Clang instead of GCC?
guix build -f guix.scm 
  --with-c-toolchain=guile@3.0.99-git=clang-toolchain

# What about that under-tested configure flag?
guix build -f guix.scm 
  --with-configure-flag=guile@3.0.99-git=--disable-networking

Handy!

Level 2: The repository as a channel

We now have a Git repository containing (among other things) a package
definition. Can’t we turn it into a
channel?
After all, channels are designed to ship package definitions to users,
and that’s exactly what we’re doing with our guix.scm.

Turns out we can indeed turn it into a channel, but with one caveat: we
must create a separate directory for the .scm file(s) of our channel
so that guix pull doesn’t load unrelated .scm files when
someone pulls the channel—and in Guile, there are lots of them! So
we’ll start like this, keeping a top-level guix.scm symlink for the
sake of guix shell:

mkdir -p .guix/modules
mv guix.scm .guix/modules/guile-package.scm
ln -s .guix/modules/guile-package.scm guix.scm

To make it usable as part of a channel, we
need to turn our guix.scm file into a
module:
we do that by changing the use-modules form at the top to a
define-module form. We also need to actually export a package
variable, with define-public, while still returning the package value
at the end of the file so we can still use guix shell and guix build -f guix.scm. The end result looks like this (not repeating things that
haven’t changed):

(define-module (guile-package)
  #:use-module (guix)
  #:use-module (guix git-download)   ;for ‘git-predicate’
  …)

(define-public guile
  (package
    (name "guile")
    (version "3.0.99-git")                          ;funky version number
    …))

;; Return the package object define above at the end of the module.
guile

We need one last thing: a .guix-channel
file
so Guix knows where to look for package modules in our repository:

;; This file lets us present this repo as a Guix channel.

(channel
  (version 0)
  (directory ".guix/modules"))  ;look for package modules under .guix/modules/

To recap, we now have these files:

.
├── .guix-channel
├── guix.scm → .guix/modules/guile-package.scm
└── .guix
    └── modules
       └── guile-package.scm

And that’s it: we have a channel! (We could do better and support
channel
authentication
so users know they’re pulling genuine code. We’ll spare you the details
here but it’s worth considering!) Users can pull from this channel by
adding it to
~/.config/guix/channels.scm,
along these lines:

(append (list (channel
                (name 'guile)
                (url "https://git.savannah.gnu.org/git/guile.git")
                (branch "main")))
        %default-channels)

After running guix pull, we can see the new package:

$ guix describe
Generation 264  May 26 2023 16:00:35    (current)
  guile 36fd2b4
    repository URL: https://git.savannah.gnu.org/git/guile.git
    branch: main
    commit: 36fd2b4920ae926c79b936c29e739e71a6dff2bc
  guix c5bc698
    repository URL: https://git.savannah.gnu.org/git/guix.git
    commit: c5bc698e8922d78ed85989985cc2ceb034de2f23
$ guix package -A ^guile$
guile   3.0.99-git      out,debug       guile-package.scm:51:4
guile   3.0.9           out,debug       gnu/packages/guile.scm:317:2
guile   2.2.7           out,debug       gnu/packages/guile.scm:258:2
guile   2.2.4           out,debug       gnu/packages/guile.scm:304:2
guile   2.0.14          out,debug       gnu/packages/guile.scm:148:2
guile   1.8.8           out             gnu/packages/guile.scm:77:2
$ guix build guile@3.0.99-git
[…]
/gnu/store/axnzbl89yz7ld78bmx72vpqp802dwsar-guile-3.0.99-git-debug
/gnu/store/r34gsij7f0glg2fbakcmmk0zn4v62s5w-guile-3.0.99-git

That’s how, as a developer, you get your software delivered directly into
the hands of users! No intermediaries, yet no loss of transparency and
provenance tracking.

With that in place, it also becomes trivial for anyone to create Docker
images, Deb/RPM packages, or a plain tarball with guix pack:

# How about a Docker image of our Guile snapshot?
guix pack -f docker -S /bin=bin guile@3.0.99-git

# And a relocatable RPM?
guix pack -f rpm -R -S /bin=bin guile@3.0.99-git

Bonus: Package variants

We now have an actual channel, but it contains only one package. While
we’re at it, we can define package
variants
in our guile-package.scm file, variants that we want to be able to
test as Guile developers—similar to what we did above with
transformation options. We can add them like so:

;; This is the ‘.guix/modules/guile-package.scm’ file.

(define-module (guile-package)
  …)

(define-public guile
  …)

(define (package-with-configure-flags p flags)
  "Return P with FLAGS as addition 'configure' flags."
  (package/inherit p
    (arguments
     (substitute-keyword-arguments (package-arguments p)
       ((#:configure-flags original-flags #~(list))
        #~(append #$original-flags #$flags))))))

(define-public guile-without-threads
  (package
    (inherit (package-with-configure-flags guile
                                           #~(list "--without-threads")))
    (name "guile-without-threads")))

(define-public guile-without-networking
  (package
    (inherit (package-with-configure-flags guile
                                           #~(list "--disable-networking")))
    (name "guile-without-networking")))


;; Return the package object defined above at the end of the module.
guile

We can build these variants as regular packages once we’ve pulled the
channel. Alternatively, from a checkout of Guile, we can run a command like
this one from the top level:

guix build -L $PWD/.guix/modules guile-without-threads

Level 3: Setting up continuous integration

This channel becomes even more interesting once we set up continuous
integration (CI).
There are several ways to do that.

You can use one of the mainstream continuous integration tools, such as
GitLab-CI. To do that, you need to make sure you run jobs in a Docker
image or virtual machine that has Guix installed. If we were to do that
in the case of Guile, we’d have a job that runs a shell command like
this one:

guix build -L $PWD/.guix/modules guile@3.0.99-git

Doing this works great and has the advantage of being easy to achieve on
your favorite CI platform.

That said, you’ll really get the most of it by using
Cuirass, a CI tool designed for and
tightly integrated with Guix. Using it is more work than using a hosted
CI tool because you first need to set it up, but that setup phase is
greatly simplified if you use its Guix System
service.
Going back to our example, we give Cuirass a spec file that goes like
this:

;; Cuirass spec file to build all the packages of the ‘guile’ channel.
(list (specification
        (name "guile")
        (build '(channels guile))
        (channels
         (append (list (channel
                         (name 'guile)
                         (url "https://git.savannah.gnu.org/git/guile.git")
                         (branch "main")))
                 %default-channels))))

It differs from what you’d do with other CI tools in two important ways:

• Cuirass knows it’s tracking two channels, guile and guix.
  Indeed, our own guile package depends on many packages provided by
  the guix channel—GCC, the GNU libc, libffi, and so on. Changes to
  packages from the guix channel can potentially influence our
  guile build and this is something we’d like to see as soon as
  possible as Guile developers.
• Build results are not thrown away: they can be distributed as
  substitutes
  so that users of our guile channel transparently get pre-built
  binaries!

From a developer’s viewpoint, the end result is this status
page listing evaluations: each
evaluation is a combination of commits of the guix and guile
channels providing a number of jobs—one job per package defined in
guile-package.scm times the number of target architectures.

As for substitutes, they come for free! As an example, since our
guile jobset is built on ci.guix.gnu.org, which runs guix publish
in addition to Cuirass, one automatically gets substitutes for guile
builds from ci.guix.gnu.org; no additional work is needed for that.

Bonus: Build manifest

The Cuirass spec above is convenient: it builds every package in our
channel, which includes a few variants. However, this might be
insufficiently expressive in some cases: one might want specific
cross-compilation jobs, transformations, Docker images, RPM/Deb
packages, or even system tests.

To achieve that, you can write a
manifest.
The one we have for Guile has entries for the package variants we
defined above, as well as additional variants and cross builds:

;; This is ‘.guix/manifest.scm’.

(use-modules (guix)
             (guix profiles)
             (guile-package))   ;import our own package module

(define* (package->manifest-entry* package system
                                   #:key target)
  "Return a manifest entry for PACKAGE on SYSTEM, optionally cross-compiled to
TARGET."
  (manifest-entry
    (inherit (package->manifest-entry package))
    (name (string-append (package-name package) "." system
                         (if target
                             (string-append "." target)
                             "")))
    (item (with-parameters ((%current-system system)
                            (%current-target-system target))
            package))))

(define native-builds
  (manifest
   (append (map (lambda (system)
                  (package->manifest-entry* guile system))

                '("x86_64-linux" "i686-linux"
                  "aarch64-linux" "armhf-linux"
                  "powerpc64le-linux"))
           (map (lambda (guile)
                  (package->manifest-entry* guile "x86_64-linux"))
                (cons (package
                        (inherit (package-with-c-toolchain
                                  guile
                                  `(("clang-toolchain"
                                     ,(specification->package
                                       "clang-toolchain")))))
                        (name "guile-clang"))
                      (list guile-without-threads
                            guile-without-networking
                            guile-debug
                            guile-strict-typing))))))

(define cross-builds
  (manifest
   (map (lambda (target)
          (package->manifest-entry* guile "x86_64-linux"
                                    #:target target))
        '("i586-pc-gnu"
          "aarch64-linux-gnu"
          "riscv64-linux-gnu"
          "i686-w64-mingw32"
          "x86_64-linux-gnu"))))

(concatenate-manifests (list native-builds cross-builds))

We won’t go into the details of this manifest; suffice to say that it
provides additional flexibility. We now need to tell Cuirass to build
this manifest, which is done with a spec slightly different from the
previous one:

;; Cuirass spec file to build all the packages of the ‘guile’ channel.
(list (specification
        (name "guile")
        (build '(manifest ".guix/manifest.scm"))
        (channels
         (append (list (channel
                         (name 'guile)
                         (url "https://git.savannah.gnu.org/git/guile.git")
                         (branch "main")))
                 %default-channels))))

We changed the (build …) part of the spec to '(manifest ".guix/manifest.scm") so that it would pick our manifest, and that’s
it!

Wrapping up

We picked Guile as the running example in this post and you can see the
result here:

• .guix-channel;
• .guix/modules/guile-package.scm
  with the top-level guix.scm symlink;
• .guix/manifest.scm.

These days, repositories are commonly peppered with dot files for
various tools: .envrc, .gitlab-ci.yml, .github/workflows,
Dockerfile, .buildpacks, Aptfile, requirements.txt, and whatnot.
It may sound like we’re proposing a bunch of additional files, but in
fact those files are expressive enough to supersede most or all of
those listed above.

With a couple of files, we get support for:

• development environments (guix shell);
• pristine test builds, including for package variants and for
  cross-compilation (guix build);
• continuous integration (with Cuirass or with some other tool);
• continuous delivery to users (via the channel and with pre-built
  binaries);
• generation of derivative build artifacts such as Docker images or
  Deb/RPM packages (guix pack).

At the Guix headquarters, we’re quite happy about the result. We’ve
been building a unified tool set for reproducible software deployment;
this is an illustration of how you as a developer can benefit
from it!

Acknowledgments

Thanks to Attila Lendvai, Brian Cully, and Ricardo Wurmus for providing
feedback on an earlier draft of this post.

About GNU Guix

GNU Guix is a transactional package manager and
an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through Guile
programming interfaces and extensions to the
Scheme language.

After years of speculation, leaks, rumors, setbacks, and rumblings of amazing behind-the-scenes demos, Apple has made its plans for a mixed reality platform and headset public. Vision Pro is “the first Apple Product you look through, not at,” Apple’s Tim Cook said, a “new AR platform with a new product” that augments reality by seamlessly blending the real world with the digital world. The headset will start at $3,499 and be available early next year. That puts the device in an entirely different class than most existing VR headsets, including the $550 PSVR2 (which requires a tethered PS5 to use) and the $500 Quest 3 that was just announced for a fall release. The technology on display here is amazing, but the presentation itself, including Apple’s proposed use cases, were thoroughly dystopian. When you’re wearing it, a video feed of your eyes can be shown on the outside display when talking to someone next to you, which looks like pure nightmare fuel to me. Apple also showed a birthday party where the dad was wearing this thing while his daughter and her friends were blowing out the candles – which, as a dad… Just no. Don’t wear the creepy glowing robot face during your daughter’s birthday party. Other than that, since it has no controllers, the gaming proposition consisted of regular “2D” games projected on a screen, so you can’t play popular VR games like Beat Saber or Gorilla Tag. Since the device tries very hard to mimic a traditional user interface in VR, many of the renders shown off during the presentation consisted of floating windows. Videoconferencing consisted of floating windows with camera feeds from the participants, for instance, while the VR user’s face is rendered onto an avatar. Showing multiple application windows floating around you definitely looks very cool, but whether or not that’s actually a pleasant user experience? I don’t know. But the biggest problem with the whole presentation is that Apple has not actually showed off anything tangible. Everything shown off during the keynote was fake – prerendered special effects layered onto video, and since nobody has received any hands-on time with the actual hardware, and thus nobody outside of Apple has seen the real user interface in action, we actually have no idea how it will actually look, feel, and perform. This is the AR/VR equivalent of using prerendered cinematics to create hype for a video game, and we should know better by now. If there’s one company that can convince people to spend $3500 to strap an isolating dystopian glowing robot mask onto their faces it’s Apple, but I still have a hard time believing this is what people want.