Created page with "My mandate was and is to provide exploratory testing and browser test automation. QA testing is a service provided to software development projects. Software development te..."
New page
My mandate was and is to provide exploratory testing and browser test automation.
QA testing is a service provided to software development projects. Software development teams have a QA testing practice for the same reason that hockey teams have goalies. You can play hockey without a goalie, but the chance of the other team scoring is a lot higher.
QA testing serves software development projects, and software development projects serve the goals of the Foundation.
Testing improves the value of these projects: https://strategy.wikimedia.org/wiki/Product_Whitepaper#Product_priority_recommendations
Software development is only a part of the Wikipedia movement. Of the goals of the movement, QA testing serves to help stabilize infrastructure and also to encourage innovation. https://strategy.wikimedia.org/wiki/Strategic_Plan/Movement_Priorities#Goals
Initial challenges:
* Earn the trust of software development projects by providing value and improving quality
* Create and maintain useful test environments
* Provide guidance and documentation of good testing practice, both exploratory/manual testing and automated testing
===Test environments===
Test environments are a precondition for testing.
Then:
* Most testing was done in production, mostly reacting to user reports of issues
* testwiki/test2wiki were underused and crufty
* beta labs was a hot mess under the covers, unreliable and unmaintainable
Now:
* test2 is fully utilized as a target for automated regression tests, run in a timely manner in the course of regular deployment
* much of the cruft on test2 has been addressed
* beta labs is puppetized, maintainable, and communicates with git
* beta labs hosts the testable AFTv5 code, primarily as a target for automated tests
Future:
* beta labs to host testable MobileFrontend
* beta labs better served by Jenkins
* policy changes so beta will serve development projects better (i.e. find a way to deploy useful extension code without merging to master)
Lessons learned:
* Reliable test environments require investment and maintenance
* Policies governing what is and is not deployed to test environments are difficult and may need to shift over time
Input wanted:
* More focus and more support on beta labs, more extensions supported and configured. MobileFrontend is underway, but we could use more.
** Database updates are particularly thorny right now. These are always done manually in production deployments, and beta is languishing. Ideally this could be done as part of the move to a more DevOps/Continuous Deployment deployment style.
* We really need a policy and infrastructure that will support experimental features and extensions, the way forward is not clear right now.
Three month timeline:
* MobileFrontend on beta labs
* Fix test2wiki so that PageTriage works so we can get some automated tests for NewPagesFeed workflow. https://bugzilla.wikimedia.org/show_bug.cgi?id=44065
* Work on supporting E3 in beta labs
* Work on supporting experimental versions of AFT on beta labs
* Possibly get more deploy builds into Jenkins and out of daemon/cron jobs
===Exploratory/human/manual testing===
Then:
* no organized or dedicated testing
* no testing community
* existing test plans were scattered and not well considered
* little or no community outreach or communication
Now:
* early Proof Of Concept to test rapid deployment didn't work too well, mandate was too broad and too complicated
** POC was in conjuction with Weekend Testers America group and Telerik Test Summit conference
* early POC to test AFTv5 was successful, important issues identified that influenced the course of the project
** POC was in conjunction with OpenHatch and involved people from Weekend Testers who remained excited about the exercise
* hiring Quim Gil puts community testing forward with scheduled events and Groups
Future:
* Build the testing community, continue to uncover issues in software and features
* Make the community self sustaining to the extent possible
* Create a framework within which others may contribute besides just testing, for example volunteer-organized test events, volunteer test plans, bug management in particular areas.
Lessons learned:
* Test events require narrow focus and clear intent
* Scheduling test events is not trivial. Software development projects shift priorities frequently and stakeholders have misunderstandings and conflicting goals.
* Adequate test environments are not always available and sometimes require preliminary work and investment.
* "Community is harder than most people think." -Marlena Compton, formerly of Mozilla WebQA, in a private conversation
Input wanted:
* SMEs to be involved in guiding test efforts
* Outreach to potential volunteer testers, particularly from the Wikipedia community (as opposed to the greater software testing community)
Three month timeline:
* Test event for AFT
* Test event for Echo
* Test event for Mobile
* Community outreach and marketing.
* Move testing Groups out of "Proposals"
===Browser test automation===
Then:
* at least one browser test automation failed after significant investment
* tools and practice were primitive and very expensive
* other browser test initiatives were scattershot, not maintainable, not standard, used inferior tools and practices
Now:
* significant research into best available tools and practice
** Test Automation Bazaar and Telerik Test Summit conferences were invaluable
* proof of concept made public in github, sparked discussion in the community (best example: http://watirmelon.com/2012/06/22/rspec-page-objects-and-user-flows/)
* hired Zeljko and the project took off. Bottom line: our browser tests find issues. Regression problems with UW and cross-browser issues found when testing IE versions are the biggest success so far
** economical and inexpensive hosted services is the key
** scalable with Jenkins on cloudbees and Sauce Labs
** good reporting
** maintainable architecture
** support for mobile
** works with our deployment practice, not against it
* our framework is best-of-breed, absolutely world-class
Future:
* More test coverage! More features tested, more depth to existing tests
* Build up the backlog of tests to be automated
** Contributions from extensions people, product people, community
** Cucumber opens a communication channel between automaters and non-programmers
* Code contributions from programmers
Lessons learned:
* Expertise in the area is required. Starting browser test automation is easy, maintainance less so.
* Good tools and good practice attract interest
* Expanding test coverage takes significant time and effort
* Automated tests have shown that we don't support IE very well
Input wanted:
* Test scenarios. I would like to have a large backlog of tests to be automated.
* Developer interest in creating browser tests
* Community contributions from competent browser test developers
Three month timeline:
* First Visual Editor basic test (we don't want to invest too much here since VE is subject to radical changes, but we want to be able to move quickly as VE matures)
* First PageTriage test (depends on test environment timeline above)
* First test to use the API, probably for delete/restore test
* Check for javascript errors opening pages
* More test scenarios in the backlog!
Far future:
It is a historical accident that software testing is erroneously conflated with Quality Assurance in the software development arena. Testing is not QA.
Software testing is "...a process of gathering information about it with the intent that the information could be used for some purpose." (-Gerald Weinberg) It is a process of investigation and reporting.
Quality Assurance, on the other hand, is methodology work. It is the examination of processes for the purpose of improving those processes.
As testing becomes more trusted and more routine, I would like to spend more effort improving software development process itself rather that only just investigating software behavior.