The documentation says I am expecting the correct behavior:

https://github.com/rspec/rspec-core/blob/master/lib/rspec/core/pending.rb?source=cc#L50-L57

Cucumber output if the step is not marked pending:

$ bundle exec cucumber features/font_selection.feature:23
Using the default profile...
............F-

(::) failed steps (::)

expected: "sans-serif"

got: "OpenDyslexic,sans-serif" (using ==) (RSpec::Expectations::ExpectationNotMetError)

./features/step_definitions/font_selection_steps.rb:15:in `/^the active content font must be the same as font prior to the preview$/'
features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview'

Failing Scenarios:
cucumber features/font_selection.feature:23 # Scenario: Discarding live preview of content font

1 scenario (1 failed)
9 steps (1 failed, 1 skipped, 7 passed)
1m12.158s

If the step is marked as pending, the default Cucumber formatter (progress) does not give a lot of information:

$ bundle exec cucumber features/font_selection.feature:23
Using the default profile...
............P-

(::) pending steps (::)

features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview'

1 scenario (1 pending)
9 steps (1 skipped, 1 pending, 7 passed)
0m55.161s

But formatter called "pretty" gives more information (... 'bug #56081 (Cucumber::Pending)' ...):

$ bundle exec cucumber -f pretty features/font_selection.feature:23
Using the default profile...
@commons.wikimedia.beta.wmflabs.org @login @reset-preferences-after
Feature: Font selection

In order to have better using experience,
As a reader and writer,
I want to change or disable the fonts for interface and content.

In addition the user is provided live preview feature: changes are applied
immediately when selection is made. Changes can either be applied or discarded
for easy testing.

Background:                                    # features/font_selection.feature:12
  Given I am logged in                         # features/step_definitions/common_steps.rb:8
  And I set "German" as the interface language # features/step_definitions/common_steps.rb:12
  And I open ULS                               # features/step_definitions/font_selection_steps.rb:1
  And I open display settings                  # features/step_definitions/font_selection_steps.rb:5
  When I open fonts panel of language settings # features/step_definitions/font_selection_steps.rb:9

Scenario: Discarding live preview of content font                                 # features/font_selection.feature:23
  When I select "OpenDyslexic" font for the content language for the live preview # features/step_definitions/panel_steps.rb:38
  And I close the panel to discard the changes                                    # features/step_definitions/panel_steps.rb:43
  Then the active content font must be the same as font prior to the preview      # features/step_definitions/font_selection_steps.rb:13
    bug #56081 (Cucumber::Pending)
    ./features/step_definitions/font_selection_steps.rb:14:in `/^the active content font must be the same as font prior to the preview$/'
    features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview'
  And the selected content font must be "system"                                  # features/step_definitions/font_selection_steps.rb:19

1 scenario (1 pending)
9 steps (1 skipped, 1 pending, 7 passed)
1m3.638s

But if change the expectation to the example from rspec documentation (so the expectation inside the pending block no longer fails:

pending('bug #56081') do

		true.should be(true)

end

I was expecting the scenario to fail, but it did not fail:

$ bundle exec cucumber -f pretty features/font_selection.feature:23
Using the default profile...
@commons.wikimedia.beta.wmflabs.org @login @reset-preferences-after
Feature: Font selection

In order to have better using experience,
As a reader and writer,
I want to change or disable the fonts for interface and content.

In addition the user is provided live preview feature: changes are applied
immediately when selection is made. Changes can either be applied or discarded
for easy testing.

Background:                                    # features/font_selection.feature:12
  Given I am logged in                         # features/step_definitions/common_steps.rb:8
  And I set "German" as the interface language # features/step_definitions/common_steps.rb:12
  And I open ULS                               # features/step_definitions/font_selection_steps.rb:1
  And I open display settings                  # features/step_definitions/font_selection_steps.rb:5
  When I open fonts panel of language settings # features/step_definitions/font_selection_steps.rb:9

Scenario: Discarding live preview of content font                                 # features/font_selection.feature:23
  When I select "OpenDyslexic" font for the content language for the live preview # features/step_definitions/panel_steps.rb:38
  And I close the panel to discard the changes                                    # features/step_definitions/panel_steps.rb:43
  Then the active content font must be the same as font prior to the preview      # features/step_definitions/font_selection_steps.rb:13
    Expected pending 'bug #56081' to fail. No Error was raised. No longer pending? (Cucumber::Pending)
    ./features/step_definitions/font_selection_steps.rb:14:in `/^the active content font must be the same as font prior to the preview$/'
    features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview'
  And the selected content font must be "system"                                  # features/step_definitions/font_selection_steps.rb:19

1 scenario (1 pending)
9 steps (1 skipped, 1 pending, 7 passed)
0m59.764s

I think the problem is that we are using Cucumber::Pending instead of RSpec::Pending, and looks like Cucumber does not behave like RSpec.

I am not sure how to call RSpec#pending from Cucumber, I have tried this:

RSpec::Core::Pending.pending('bug #56081')

but no luck:

uninitialized constant RSpec::Core (NameError)

Dan, any idea on how to make this happen?

We could reimplement #pending in a world helper module and rescue/re-raise the Cucumber::Pending exception as something else.

module MediawikiSelenium::Pending

class Failure < StandardError; end

def pending(*args)
  super
rescue Cucumber::Pending => e
  if block_given?
    raise Failure, e.message
  else
    raise e
  end
end

end

World(MediawikiSelenium::Pending)

I think just my Ruby-fu is failing me. See comments 6 and 7. RSpec::Pending already does what I want, but looks like I am calling Cucumber::Pending here:

pending('bug #56081') do

true.should be(true)

end

How do I make sure RSpec::Pending is called. See comment 7 on what I have tried.

I'm not sure that cucumber even uses rspec-core. It might just use the matchers. Either way, I think implementing our own helper is more inline with the Cucumber Way—not that I think the Cucumber Way is great, but it's better not to wrestle with it too much.

If you don't want to override the behavior of #pending globally, then we can use a different name (#pending_or_fail, #temporarily_pending, etc.).

As far as I understand it, Cucumber does use rspec, that is why we explicitly use rspec-expectations[2].

1: https://github.com/wikimedia/mediawiki-selenium/blob/master/mediawiki_selenium.gemspec#L26

Right, but rspec-expectations is separate from rspec-core which is what implements describe, it, pending, etc.

I see, that was what I misunderstood. Thanks. :)

Ok then, so we could do three things:

#1 patch Cucumber::Pending

#2 require RSpec::Core::Pending and use it

#3 report a bug for Cucumber::Pending saying it does not behave as RSpec::Core::Pending, which is confusing

I think I'd still rather just reimplement pending for our own Cucumber World (see comment 9) over monkey patching Cucumber::Pending (#1) or adding another dependency (#2).

(In reply to Dan Duvall from comment #15)

I think I'd still rather just reimplement pending for our own Cucumber World
(see comment 9) over monkey patching Cucumber::Pending (#1) or adding
another dependency (#2).

Sounds good to me! :)

Change 160024 had a related patch set uploaded by Dduvall:
Stricter pending behavior for falsely passing steps

https://gerrit.wikimedia.org/r/160024

Change 160024 merged by jenkins-bot:
Stricter pending behavior for falsely passing steps

https://gerrit.wikimedia.org/r/160024

Released in 0.4.0.

Thanks Dan, I have tested this and it works great! :)