README.md

Ruby C Extensions, Explained

• Background
• How To Use This Repository
• Strategies
  • Strategy 0, "isolated"
  • Strategy 1, "system"
  • Strategy 2a, "packaged_source"
  • Strategy 2b, "packaged_tarball"
  • Strategy 3, "precompiled"
• Strategy Combinations
  • "system", fallback to "packaged"
  • "precompiled", fall back to "packaged", leave option for "system"
• FAQ
  • How do you test cross-compiling gems on Github Actions?
  • What's the significance of the flowers you're using as a background image for your RubyKaigi slides?

Background

Hello there! Welcome to Ruby C Extensions, Explained.

This repository, and the example ruby gems in it, were originally written as companion materials for some conference talks being given in 2021 by Mike Dalessio (@flavorjones).

RubyKaigi 2021 talk 2021-09-11

(Video)

This talk focuses mostly on the mechanics of the gems in this repository.

RubyConf 2021 talk 2021-11-08

(Video)

This talk covers additional material, including how to test extensions, how to think about Trust in the context of precompilation, and why Nokogiri chose to start precompiling.

How To Use This Repository

Each of the gems in this repository demonstrate a different strategy for integrating with (or packaging) third-party libraries. A brief explanation of the strategies is below, and the code for each gem is in a distinct subdirectory, along with a README that contains a more complete explanation.

Each gem builds on the previous one if you tackle them in this order:

• /isolated
• /system
• /packaged_source
• /packaged_tarball
• /precompiled

These are actual, working gems! And they've all been pushed to rubygems.org:

• gem install rcee_isolated
• gem install rcee_system
• gem install rcee_packaged_source
• gem install rcee_packaged_tarball
• gem install rcee_precompiled

Or they can all be installed as dependencies of the meta-gem rcee:

• gem install rcee

Also note that these gems are tested. In particular, the precompiled native gems have a rather complex setup to build, install, and test across platforms using rake-compiler-dock and Github Actions. See the Actions running at https://github.com/flavorjones/ruby-c-extensions-explained/actions

Strategies

A few commonly-followed approaches have emerged in the Ruby ecosystem. Broadly categorized, the strategies are:

Strategy 0, "isolated"

A self-contained C extension (no third-party libraries).

See subdirectory /isolated.

Real-world examples:

• https://github.com/bcrypt-ruby/bcrypt-ruby

Strategy 1, "system"

Find and use an third-party library already installed on the target system.

See subdirectory /system.

Real-world examples:

• https://github.com/rmagick/rmagick
• https://github.com/sparklemotion/sqlite3-ruby

Strategy 2a, "packaged_source"

Package the source code for the third-party library, and compile it into the C extension during installation.

See subdirectory /packaged_source.

Real-world examples:

• https://github.com/ruby/psych
• https://github.com/rubys/nokogumbo
• https://github.com/sass/sassc-ruby

Strategy 2b, "packaged_tarball"

Package the tarball distribution for the third-party library, use its configuration toolchain and build a library that the C extension can link against.

See subdirectory /packaged_tarball.

Real-world examples:

• https://github.com/sparklemotion/nokogiri
• https://github.com/kwilczynski/ruby-magic

Strategy 3, "precompiled"

Precompile the library and C extension ahead-of-time for a subset of platforms; fall back to packaged_tarball strategy for other platforms.

See subdirectory /precompiled

Real-world examples:

• https://github.com/sparklemotion/nokogiri
• https://github.com/kwilczynski/ruby-magic
• https://github.com/grpc/grpc/tree/master/src/ruby

Strategy Combinations

These strategies can also be combined!

"system", fallback to "packaged"

Real-world examples:

• https://github.com/ruby/psych

"precompiled", fall back to "packaged", leave option for "system"

Real-world examples:

• https://github.com/sparklemotion/nokogiri
• https://github.com/kwilczynski/ruby-magic

FAQ

I'll try to answer questions asked during conference Q&A here.

How do you test cross-compiling gems on Github Actions?

Nokogiri uses the rake-compiler-dock OCI images directly to build the gem; and then uses the upload-artifact/download-artifact actions to install that gem on the target system.

See https://github.com/sparklemotion/nokogiri/blob/main/.github/workflows/gem-install.yml for the concrete implementation!

What's the significance of the flowers you're using as a background image for your RubyKaigi slides?

I tweeted a brief teaser video in which I explained the joke!

When I edited my RubyKaigi talk about C extensions, some bits had to be cut for time. Here's a fun teaser that I put together from some of those bits. [video link] pic.twitter.com/wZjSepHNpq

— mike dalessio (@flavorjones) September 9, 2021

In short, my friend Mark once sent me flowers for my birthday along with a card that read:

Fetching nokogiri-1.8.0.gem (100%)
Building native extensions. This could take a while...

Mark is funny. This was the inspiration for the talk's title.