QA Engineer new to SpecFlow and BDD. How will/can BDD & SpecFlow help me with UAT and test automation?

[Eric Bates]

Hi everyone, I am new to these tools/features and I am trying to get a good
understanding of them. Please correct me if anything I say is wrong.
From what I have learned a PM or BA will write user stories in a BDD
format. Is there a tool for doing this, or will this just be done by hand
in a TXT file?

Then a developer will take the BDD formatted user story and import it into
Specflow???
SpecFlow then translates the BDD formatted user story into test code
fragments which the developer will use to run Unit Tests against the
application code. Am I correct here?

As a QA Engineer how will my company's transition to using BDD help me?
I have been given a goal of using BDD to develop UI and application level
test automation.
I have become rather lost at this point and how to accomplish this goal.

All information and comments greatly appreciated.

[Dan Ryan]

Hi Eric,

Because I do not know anything about where you work and your team setup, I cannot really advise you in detail.  What I can do is describe the way that BDD has been leveraged at a number of places that i have worked.

I think it is important to understand that specflow feature files are both an output to the requirements phase and are also used to test that the requirements have been met (thereby automating the UAT phase).  For developers the feature files serve a third purpose: driving development, test first, through the practice of Acceptance Test Driven Development ATDD.

Because of the multi purpose nature of the scenario features, it is important that all parties (business, BA's, developers, QA, etc) are satisfied with the structure, scope and vocabulary of the features.  Although this can be done independently, and then amended and approved by the various parties; my recommendation would be to get a small group and work on these features as a team. I find this approach works well as you spend less time handing over and amending scenarios as part of a review cycle. I also find that talking through the scenarios in a group stimulates discussion around edge cases that might have otherwise been missed.  Where I have worked, QA have taken a lead role in this process.

Once the features files have been written and agreed by the various parties, they can be used to drive development.  This is done by writing code that performs each operation described by each scenario step.  Because we are using the scenarios to drive development, it is best that this is performed by the development team.  If this is unclear I recommend googling the agile term: YAGNI.  When done properly, what you are left with are automated tests that prove that not only does the application work, but that it also meets the requirements (that were possible to automate) as specified by the acceptance criteria. Pretty powerful don't you agree?

I recommend that these tests are automatically run, preferably whenever the code changes, using Continuous Integration software such as Teamcity. This ensures that regression issues are highlighted at the earliest opportunity.

Because you are left with automated tests, as a QA expert you should have more time for the following: exploratory testing, working on the definition of additional feature scenarios, integration testing (i.e. ensuring that the individual features work as a coherent system) and performance testing.

I hope this is helpful? Let me know if you have any further questions? 

Dan

[Eric Bates]

Hi Dan,
Thanks for the feed back.
I am currently in a small software team of 8 developers, and 3 QA.
(including me.)
We are creating applications in C# with VS 2012.

The company has been around for a while, but there has been a major gutting
of the Eng dept. and almost everyone here now is new to the company.
It have been decided that as an Eng Dept. we are going to follow the BDD
process.
We are using JIRA and Greenhopper as our main "Agile"
command/control/repository.

For the Feature files, Are there any good tools for creating these or will
the BA/PMs be stuck using notepad?
I get that SpecFlow will use the feature files to generate "unit" tests.
And that we can set up all these tests to auto run as part of our
continuous integration build process. We are going to use Bamboo.

What I am trying to figure out is, How can these feature files help QA in
writing user level automation tests.
In previous companies I have used tools like HP QTP or Selenium. Can /
should I still be using these tools under the BDD process?
Do you know of a way to link automation tools like these to the feature
files?

[Umair Chagani]

Hi Eric,

Maybe I can shed some light on this process.

Regarding the tools used to create the feature files: Notepad is perfectly
fine for writing these. Of course, if the BAs have access to Visual Studio
and SpecFlow, it will make writing the feature files more intuitive.
However, if you are in the same boat as most, VS licenses are restricted
to the dev team. What you can do is have the BAs use Notepad++ (free) and
use the instructions
here<http://devtnt.techtalk.at/2011/06/devtnt-v21-notepadspecflow-syntax.html>
to
get the same benefits as writing in VS.

I would also like to add that one of the key components of BDD is a
consistent DSL. This can grow organically and should be a collaboration
between BA/QA/Dev. If you can get the BAs and QAs to pair up when writing
the scenarios, the chances of success increases exponentially.

Regarding automation of acceptance testing: You should be using the
feature files for your automation. If the devs are using SpecFlow for
their unit testing, then I would highly recommend that both you and the
devs use the SAME feature files (and of course ensure they are version
controlled). This will also give you a great way to trace your automation
tests to unit tests to production code.

The primary benefit of using BDD and gherkin is that all parts of the
engineering team are on the same page. BA/QAs write the scenarios. Devs
use the scenarios for TDD, Automation uses the scenarios for UAT.
Additionally, by implementing BDD correctly, you've created a
self-documenting system that is always up to date. If it's not up to date,
your unit tests and your automation tests will break. This aids
tremendously in continuous integration by giving quick feedback to the devs.


I hope this helps.

Regards,
Umair

Post by Eric Bates
Hi Dan,
Thanks for the feed back.
I am currently in a small software team of 8 developers, and 3 QA.
(including me.)
We are creating applications in C# with VS 2012.
The company has been around for a while, but there has been a major
gutting of the Eng dept. and almost everyone here now is new to the company.
It have been decided that as an Eng Dept. we are going to follow the BDD
process.
We are using JIRA and Greenhopper as our main "Agile"
command/control/repository.
For the Feature files, Are there any good tools for creating these or will
the BA/PMs be stuck using notepad?
I get that SpecFlow will use the feature files to generate "unit" tests.
And that we can set up all these tests to auto run as part of our
continuous integration build process. We are going to use Bamboo.
What I am trying to figure out is, How can these feature files help QA in
writing user level automation tests.
In previous companies I have used tools like HP QTP or Selenium. Can /
should I still be using these tools under the BDD process?
Do you know of a way to link automation tools like these to the feature
files?
--
You received this message because you are subscribed to the Google Groups "SpecFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[Dan Ryan]

We are using visual studio to construct the feature files. This is either
done in a group (BA's, QA, DEV) sitting around a projector or by either a
QA or Developer on there local machine and then sent to the relevant
parties for feedback and approval. In the places that I have worked the
BA's were not able to construct features files in a way that was suitable
for automation. Hence why this has become a whole team approach.
Incidently the BA's where I have worked have been really good at reviewing
the features against the requirements and highlighting any misunderstanding
and missing requirements. I would be interested to hear other peoples
opinions on who should be responsible for authoring feature files?

As mentioned in my previous post, I believe that developers are best placed
to perform what you call "user level automation tests". The reason I
believe this is that developers should be best placed to do the automation
(which is coding after all) and they should also be using the tests to
drive development. Where you might be confused is the assumption that
features scenarios will be used to by developers to perform "Unit" tests.
Now the language used within the industry is a bit confused in relation to
what a unit test is but the type of testing that I would expect developers
to be performing is not the type of unit testing that issolates individual
components in the system (i.e. just test a particular class and not it's
dependent classes) but what I would call and integrated test which avoids
isolating individual components and tests as much of the system end-to-end
as possible. When a developer has finished implementing a particular
scenario and all the tests pass, you should have a high degree of
confidence that the code is functionally correct and that it also meets all
the defined and automatable acceptance criteria. In my mind this should
make up the balk of the automation testing necessary for a particular
project. However you might want to consider creating some automation
scripts for things like testing a partcular environment is configured
correctly or some automated integration testing that checks the
relationship between the different feature scenarios; I would not recommend
Specflow as a suitable tool for this type of automation.

One final point. If you are developing an application that has a UI, you
(as a team) will need whether you bind the scenarios at the UI level (i.e.
using a tool such as selenium) or just below this i.e. at the controller
level if using the MVC pattern or the view model if using the MVVM
pattern. I would strongly recommend the later, but if you do go with this
recommendation you might want to consider supplementing this with some
basic UI smoke tests.

Does this help?

Post by Eric Bates
Hi Dan,
Thanks for the feed back.
I am currently in a small software team of 8 developers, and 3 QA.
(including me.)
We are creating applications in C# with VS 2012.
The company has been around for a while, but there has been a major
gutting of the Eng dept. and almost everyone here now is new to the company.
It have been decided that as an Eng Dept. we are going to follow the BDD
process.
We are using JIRA and Greenhopper as our main "Agile"
command/control/repository.
For the Feature files, Are there any good tools for creating these or will
the BA/PMs be stuck using notepad?
I get that SpecFlow will use the feature files to generate "unit" tests.
And that we can set up all these tests to auto run as part of our
continuous integration build process. We are going to use Bamboo.
What I am trying to figure out is, How can these feature files help QA in
writing user level automation tests.
In previous companies I have used tools like HP QTP or Selenium. Can /
should I still be using these tools under the BDD process?
Do you know of a way to link automation tools like these to the feature
files?

[Eric Bates]

I think you have provided some Very good information.

I think we have a difference in terms on Unit and UAT automation.
For me UAT "User level" has traditionally been done with tools like
Selenium on a complete build in the Pre-Production environment.
Unit tests are focused on testing classes and individual components in VS.
Integration tests would be focused on more system end to end tests, prior
to handing a build over to QA for User level testing.

Most all my QA experience is on UI level testing. Both manual and Automated
(Selenium / QTP).

Let me go over how I did things before trying to implement BDD processes.
A PM would write requirements. (the team does a Req Review). Reqs are
imported into a tool like Quality Center. We (QA) would then write manual
and automated user level test cases to test and verify the Reqs.

In this new Agile / BDD process, The Req/User stories are created. From
these User Stores, we develop the acceptance criteria to verify the user
story is developed correctly. The team will then takes the Acceptance
Criteria list and turns these into "Gherkin" language feature files. Does
this sound accurate?
So, at this point if I am understand this all correctly, we will branch
off. Development team takes the "Gherkin" language feature files, Imports
into SpecFlow and starts developing the tests created from the feature
files. Then finally starts write application code.
QA would then take a copy of the "Gherkin" language feature files and link
these to Manual and Automated user level tests?? Do this sound like the
right/best process?
OR would QA not bother with the "Gherkin" language feature files, and
simply create the Manual and Automated user level tests which would link
directly to the Acceptance Criteria for the User Story.

Sorry for all the Q&A but I have been reading and researching BDD for the
past two weeks and I am a little surprised at all the information out there
but at the same time none of the info is written from a newbe perspective,
so it has been a little difficult getting my brain wrapped around this
whole new process. :-)

I hope I everything I wrote here makes sense.
I have a lot of experience in Waterfall and basic Agile SDLC, but trying to
understand the right/best way to implement BDD with Visual Studio and
SpecFlow, is a new learning experience.

[Dan Ryan]

If you were to adopt the process as I see it, the development team would
produce the automated user level acceptance tests instead of QA. One way
that this could be done is by linking scenario steps to Selenium actions
(similar to the kind that you have done in the past but now linked to
acceptance criteria). These tests if written appropriately could also be
used in different environments such as pre-prod to ensure these
environments are configured correctly.

Because features describe individual features of the application and not
necessariliy how the different features interact. You are left with a gap:
how do the individual features of the application work together to solve
the business problems. This is where you might consider writing some
Integration tests. These could be written in Selenium but would not be
tied to Specflow features. Personally I would expect QA to take ownership
of these scripts.

Now I have tried to describe this in a way that relates back to how you
have worked in the past. The way I work, the developers would implement
the tests at one level down from the UI. These should test all the code
except the UI binding, which QA could cover as part of what I described in
my previous paragraph.

What I would definately not advocate is using the same features files to
implement different tests across different teams.

Post by Eric Bates
I think you have provided some Very good information.
I think we have a difference in terms on Unit and UAT automation.
For me UAT "User level" has traditionally been done with tools like
Selenium on a complete build in the Pre-Production environment.
Unit tests are focused on testing classes and individual components in VS.
Integration tests would be focused on more system end to end tests, prior
to handing a build over to QA for User level testing.
Most all my QA experience is on UI level testing. Both manual and
Automated (Selenium / QTP).
Let me go over how I did things before trying to implement BDD processes.
A PM would write requirements. (the team does a Req Review). Reqs are
imported into a tool like Quality Center. We (QA) would then write manual
and automated user level test cases to test and verify the Reqs.
In this new Agile / BDD process, The Req/User stories are created. From
these User Stores, we develop the acceptance criteria to verify the user
story is developed correctly. The team will then takes the Acceptance
Criteria list and turns these into "Gherkin" language feature files. Does
this sound accurate?
So, at this point if I am understand this all correctly, we will branch
off. Development team takes the "Gherkin" language feature files, Imports
into SpecFlow and starts developing the tests created from the feature
files. Then finally starts write application code.
QA would then take a copy of the "Gherkin" language feature files and link
these to Manual and Automated user level tests?? Do this sound like the
right/best process?
OR would QA not bother with the "Gherkin" language feature files, and
simply create the Manual and Automated user level tests which would link
directly to the Acceptance Criteria for the User Story.
Sorry for all the Q&A but I have been reading and researching BDD for the
past two weeks and I am a little surprised at all the information out there
but at the same time none of the info is written from a newbe perspective,
so it has been a little difficult getting my brain wrapped around this
whole new process. :-)
I hope I everything I wrote here makes sense.
I have a lot of experience in Waterfall and basic Agile SDLC, but trying
to understand the right/best way to implement BDD with Visual Studio and
SpecFlow, is a new learning experience.

[Umair Chagani]

Eric,

I would take a simpler approach. There is no need for "acceptance
criteria" and feature files because your scenarios ARE your acceptance
criteria. If you do both, now your team has to maintain two sets of
criterion, each of which could change independently.

At my company we have a fairly large agile teams and the process we follow
is this:

1. BA/QA/Sometimes Devs pair up to write feature files. This is best
done, as Dan mentioned, around a projector/whiteboard.

2. Once scenarios are written, QA works on automating them, devs work on
implementing the features. Both QA and Devs should work together to ensure
that devs are adding unique IDs to their code so you can use them in your
automation (Selenium WebDriver, please don't use QTP. See why
here<http://paulhammant.com/blog/die-qtp-die.html/>).
The key here is that the automation tests are being written at the same
time (or before) the implementation is being done. Also, I would not
create "copies" of any files. Use the same file (version controlled). If
QA has their own copy, dev has their own copy, you will get yourself into a
real mess.

3. QA should *absolutely* use the feature files. EVERYONE should use the
feature files. The scenarios in the feature files are what have been
approved by the BAs. Again, I *highly* recommend you have one set up
acceptance criteria. If you are using BDD and feature files, there is no
need to maintain a separate set of acceptance criteria. We did this in my
company and we failed miserably. The BAs would change the "acceptance
criteria" but either forget or neglect to communicate the change in
acceptance criteria and the feature files were never updated with the new
criterion. So either the implementation was wrong, or the automation tests
were wrong, or both. If the "acceptance criteria" reside in one place
(remember, the scenarios in the feature files ARE your acceptance
criteria), then this problem doesn't exist. When acceptance criteria
changes (and it always does), the BA updates the feature file, and when
QA/Devs get the latest version from version control, they automatically
have the updated version. They can then proceed to implement the changes.

4. Once automation is complete, devs can run the automation tests until
they are green. Once green, that user story is complete.

Hope that helps.

Post by Eric Bates
I think you have provided some Very good information.
I think we have a difference in terms on Unit and UAT automation.
For me UAT "User level" has traditionally been done with tools like
Selenium on a complete build in the Pre-Production environment.
Unit tests are focused on testing classes and individual components in VS.
Integration tests would be focused on more system end to end tests, prior
to handing a build over to QA for User level testing.
Most all my QA experience is on UI level testing. Both manual and
Automated (Selenium / QTP).
Let me go over how I did things before trying to implement BDD processes.
A PM would write requirements. (the team does a Req Review). Reqs are
imported into a tool like Quality Center. We (QA) would then write manual
and automated user level test cases to test and verify the Reqs.
In this new Agile / BDD process, The Req/User stories are created. From
these User Stores, we develop the acceptance criteria to verify the user
story is developed correctly. The team will then takes the Acceptance
Criteria list and turns these into "Gherkin" language feature files. Does
this sound accurate?
So, at this point if I am understand this all correctly, we will branch
off. Development team takes the "Gherkin" language feature files, Imports
into SpecFlow and starts developing the tests created from the feature
files. Then finally starts write application code.
QA would then take a copy of the "Gherkin" language feature files and link
these to Manual and Automated user level tests?? Do this sound like the
right/best process?
OR would QA not bother with the "Gherkin" language feature files, and
simply create the Manual and Automated user level tests which would link
directly to the Acceptance Criteria for the User Story.
Sorry for all the Q&A but I have been reading and researching BDD for the
past two weeks and I am a little surprised at all the information out there
but at the same time none of the info is written from a newbe perspective,
so it has been a little difficult getting my brain wrapped around this
whole new process. :-)
I hope I everything I wrote here makes sense.
I have a lot of experience in Waterfall and basic Agile SDLC, but trying
to understand the right/best way to implement BDD with Visual Studio and
SpecFlow, is a new learning experience.
--
You received this message because you are subscribed to the Google Groups "SpecFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/groups/opt_out.

[Eric Bates]

Dear Dan and uchagani,
Thank you both for all your input. I think from your information I have a

[Dan Ryan]

You're most welcome.

It is interesting that uchagani and I have a different approach (although I
think we agree on the main principles). Personally I would be interested
in how other people work and collaborate on within their teams?

One final point: you might want to look at http://watin.org/ if you plan to
perform UI automation as the bindings can be done in .NET. I am not sure
if this is possible in Selenium (it wasn't when I last used it many years
ago).

Post by Eric Bates
Dear Dan and uchagani,
Thank you both for all your input. I think from your information I have a
pretty good plan of action.

[Eric Bates]

I want the QA team to develop and execute the UI level automated tests.
But the QA team does not have VS licenses, which makes trying to keep all
the code/scripts/tests stored in VS not very likely.
I think this pushes me towards Selenium over WaitN.
But I am very interested in seeing what WaitN can do and how it compares to
Selenium.
Part of my dilemma is trying to figure out how QA can fit into this whole
process of developing in VS and using Specflow but without having a VS
license.

My goal is to have the QA team contribute more as a White box test team.
There will always be a time and place for black box tester but my
preference is for my QA Engineers to gain and use white box level
skills. This skill set will make them more valuable in the future.