Ruby 1.9.2, RubyGems 1.3.7 and the disappearance of Gem.path

3 September 2010

When I upgraded to Ubuntu 10.10 Beta, my Ruby (1.9.1) and its RubyGems (and thus any loading of any external code) stopped working pretty quickly. Searching around didn’t do a bunch of help, until I stumbled upon a great entry by YAMADA Akira 4 days ago entitled “Ruby 1.9.2とRubyGems 1.3.7とGem.pathの消失”.

I’m providing a translation here for those interested, albeit a very poor one with many inaccuracies.

If you’d like a really short way to get your gem command working again, add this line to the top of your /usr/lib/ruby/1.9.1/rubygems.rb:

Gem::QuickLoader.remove unless !defined?(Gem)

The problem is that it doesn’t actually seem to fix the issue (not on Ubuntu, anyway). While gem runs and installs things, require is still broken. I’m continuing to investigate.

The source text is copyright to its author YAMADA Akira.

Following a Debian bug report (Bug#588125), I’ve discovered that the situation is looking like this:

$ gem1.9.1 list
/usr/lib/ruby/1.9.1/rubygems/source_index.rb:68:in `installed_spec_directories': undefined method `path' for Gem:Module (NoMethodError)

Ruby 1.9.2 and RubyGems 1.3.7 are both combined in the one package, but some part of RubyGems’ code (used by 1.3.7) is incompatible with Ruby 1.9.2. It seemed a curious thing that Gem.path should disappear, so I decided to take a bit of a look.

Symptoms

$ ruby1.$ ruby1.9.1 -ve 'p Gem.path'
ruby 1.9.2dev (2010-07-30) [i486-linux]
["/home/akira/.gem/ruby/1.9.1", "/usr/lib/ruby/gems/1.9.1"]

Nothing wrong here. Next.

$ ruby1.9.1 -e 'require "rubygems"; p Gem.path'
/usr/lib/ruby/1.9.1/rubygems/source_index.rb:68:in `installed_spec_directories': undefined method `path' for Gem:Module (NoMethodError)
[...]

Huh.

$ ruby1.9.1 -e 'require "rubygems"'
/usr/lib/ruby/1.9.1/rubygems/source_index.rb:68:in `installed_spec_directories': undefined method `path' for Gem:Module (NoMethodError)
[...]
    from /usr/lib/ruby/1.9.1/rubygems.rb:839:in `searcher'
    from /usr/lib/ruby/1.9.1/rubygems.rb:478:in `find_files'
    from /usr/lib/ruby/1.9.1/rubygems.rb:982:in `load_plugins'
    from /usr/lib/ruby/1.9.1/rubygems.rb:1138:in `<top>'
    from <internal:lib/rubygems/custom_require>:29:in `require'
    from <internal:lib/rubygems/custom_require>:29:in `require'
    from -e:1:in `<main>'

Oh.

$ ruby1.9.1 -e 'p Gem.path; begin; require "not exist"; rescue LoadError; end; require "rubygems"; p Gem.path'
["/home/akira/.gem/ruby/1.9.1", "/usr/lib/ruby/gems/1.9.1"]
["/home/akira/.gem/ruby/1.9.1", "/var/lib/gems/1.9.1"]

Oh, dear.

It seems like Gem.path disappears in some instances. At this stage I took the version of RubyGems included with 1.9.2 and the separately distributed version, did a diff and found that something like this does the trick:

$ ruby1.9.1 -e 'p Gem.path; Gem::QuickLoader.remove; require "rubygems"; p Gem.path'
["/home/akira/.gem/ruby/1.9.1", "/usr/lib/ruby/gems/1.9.1"]
["/home/akira/.gem/ruby/1.9.1", "/var/lib/gems/1.9.1"]

As a workaround, this is fine. But, in order to actually figure this out and to know why it works, it took a considerable amount of effort, including using -d as well as a lot of printf debugging, chasing chains of exceptions, only to find it was coming from somewhere else, and so on …

How it works

Here’s how it seems to go, after finally working it out.

Firstly, “require 'rubygems'” gets run (e.g. in the gem command), and as a result rubygems.rb is loaded. This rubygems.rb is from RubyGems 1.3.7.

In rubygems.rb, it does a “require 'rubygems/defaults/operating_system'“, only this “operating_system” doesn’t exist on the system (*). Remember that by this stage Ruby 1.9.2′s “require” has been replaced by RubyGems. Hence if rubygems/defaults/operating_system.rb (or .so) is missing, RubyGems’ own mechanism is used to see if it can try to save the situation.

In the environment at the moment when the issue occurred, RubyGems is sourced from RubyGems 1.3.7, which isn’t included in Ruby 1.9.2. Despite this, the “require” executing at the moment is from the RubyGems which is included in Ruby 1.9.2. This is from the “gem_prelude” found within the Ruby interpreter, the behaviour of which differs from the rubygems/custom_require.rb (from RubyGems 1.3.7) found on your $LOAD_PATH.

So, then Gem.try_activate is called. Gem.try_activate is actually Gem::QuickLoader.try_active, which comes from Ruby 1.9.2. This in turn calls Gem::QuickLoader.load_full_rubygems_library, and then Gem::QuickLoader.remove. That in turn removes any Gem stuff setup by gem_prelude that might cause issues later.

module QuickLoader
  @loaded_full_rubygems_library = false

  def self.remove
    return if @loaded_full_rubygems_library

    @loaded_full_rubygems_library = true

    class << Gem
      undef_method(*Gem::GEM_PRELUDE_METHODS)
    end

    remove_method :const_missing
    remove_method :method_missing

    Kernel.module_eval do
      undef_method :gem if method_defined? :gem
    end
  end
[...]
end

Now then, this “remove” was called originally all because of the internally executed “require 'rubygems'” which happened as a result of us trying to run “require 'rubygems'” in the first place!

“remove” is supposed to be used when we’re trying to load the original RubyGems, but in this case the RubyGems 1.3.7′s rubygems.rb gets loaded first. Thus, after running ‘remove’ the redefinition of Gem that we thought would definitely happen never occurs.

And thus Gem.path vanishes.

The problem of combination

This problem doesn’t happen when Ruby 1.9.2 is used on its own. The rubygems.rb included with it has the following code at the start:

gem_disabled = !defined? Gem

unless gem_disabled
  # Nuke the Quickloader stuff
  Gem::QuickLoader.remove
end

This is not included in the rubygems.rb included with RubyGems 1.3.7.

When only Ruby 1.9.2 is installed, at the time of the “require 'rubygems'” at the start of the above story, the code from gem_prelude is removed, and the entire set of RubyGems’ code is loaded in its place. Similarly, if the rubygems.rb from RubyGems 1.3.7 included the same code at the beginning, at the start of the first “require 'rubygems'” the code from gem_prelude would be removed there too.

Whatever the case, the problem starts with “require 'rubygems/defaults/operating_system'“, where the running code is actually gem_prelude’s “require“. This doesn’t change, but as the effects of “remove” only take effect once (@loaded_full_rubygems_library), hence the issue of Gem.path disappearing doesn’t occur.

(*) That’s not to say that operating_system not being present on the system is an error. By “rescue”ing the LoadError in a manner like below, the code doesn’t really care if the file is present or not.

begin
  # Defaults the operating system (or packager) wants to provide for RubyGems.
  require 'rubygems/defaults/operating_system'
rescue LoadError
end

-d option

Though not strictly related to this discussion, I realised while doing this investigation that Ruby 1.9.2′s -d option was incredibly handy:

$ ruby1.9.1 -d -e 'require "rubygems"'
[...]
/usr/lib/ruby/1.9.1/rubygems.rb:630: warning: method redefined; discarding old path
<internal:gem_prelude>:43: warning: previous definition of path was here
[...]
<internal:lib/rubygems/custom_require>:29: warning: loading in progress, circular require considered harmful - /usr/lib/ruby/1.9.1/rubygems.rb
[...]

That’s it from me. My translation skill is less, so some parts may not flow so well. Try reading the bug report if you want more info. Thanks a bunch!