The Way to OSS: Test Automation Framework "Bucky"

Hi there, my name is Jye Ruey. I'm a Software Engineer in Test (SET) from LIFULL. This article is a translation of 自動システムテストツール「Bucky」OSS化までの道のり - LIFULL Creators Blog, which is written in Japanese by Rikiya Hikimochi.

Introduction

We SET group publish a test framework "Bucky" as an Open Source Software (OSS).

github.com github.com

Bucky was a internal tool, which was used for the automation testing. There are a lot of difficulties on the way to OSS.

This article will share these difficulties and also share the knowledge we gain on the way.

Introduction
Why we decide to publish Bucky as an OSS
The way to OSS
After release
At Last

Why we decide to publish Bucky as an OSS

"Make LIFULL engineer more visible"

Although our main service "LIFULL HOME'S", a real estate and housing information aggregation web service, is well-known in Japan, LIFULL's engineers who make this service are still little-known. By publishing Bucky as an OSS, we suppose that it will make us become more famous.

"Hope everyone can use it"

Simply, we just hope more people can give Bucky a try. We use Bucky almost everyday and we think Bucky is so convenient. So we will like to share with everyone in the world.

"For more feedback from outside, and improving it become more easier to use"

Although Bucky is open now, there are still a lot of part can be improved. Because we want to make Bucky better, we will take the external feedback aggressively. And we are looking forward to getting the issue and Pull Requests.

The way to OSS

1. Refactoring

Because the OSS will release under our company's name, the code should be clean as much as we can. We refactor the code by using static analysis tool Rubocop and CodeClimate. Here are the steps:

Integrate Code Climate and Github. And analyze the code and measure the coverage of the unit test.
By the integration, create the issue detected by Code Climate on Github.
Assign the issue to SET member, and fix refactor the code.
Double check the code by using CodeClimate Cli on the local environment

By using Code Climate, it is more easier to manage the issue and becomes more efficient in refactoring.

Furthermore, Code Climate shows maintainability and coverage on the dashboard. It improves our motivation to refactor the code.

CodeClimate rank the code from E to A.

Maintainability transition of the repository.

The badges also have a good looking.

codeclimate.com

2. Adopting system testing

Improving the whole quality. We adopt the system testing. To ensure the Bucky command operation and the execution of Bucky test script, which the unit testing can't.

By introducing the test framework "Bats", we can make the output and status in Bash, as the expectation in the system testing.

github.com

The structure of system testing is as follow:

1.Github push triggers CircleCI, and run container as below.

・ container for test (Nginx)

・ selenium-standalone container

・ test execution container (Bucky)

2.With Bats, we can validate the execution output and exit code are same as expected operation. The system test script implement in here.

3. Register on RubyGems.org

Making Bucky as a gem package maybe more easier to use. We register on RubyGems.org at the same time that OSS release.

bucky-core | RubyGems.org | your community gem host

After release

Doesn't prepare the operational procedures

We want to release as fast as possible, so the entire document didn't prepare well. This will lead some problem as bellow.

Problem

User don't know how to use it after install the gem package.

We only got the simple procedures in README at that time.
The way of using Bucky-management doesn't clear.

Bucky-core is for test implementation and execution; Bucky-management is for test report. User don't know how to use it even though they read the README.

Solution 1: Prepare article for Hands-on guild

Quick start
With Bucky-management
Qiita blog hands-on article (Written in Japanese)

Solution 2: Updating README structure

Adding the setup step.

Procedure of installation and test script implementation is listed in Setup. Procedure of executing of execution is listed in Usage.

■Before

- Bucky-Core
    - Overview
    - Feature
        - Set connection infomation for database
    - Usage
        - Install
        - Run test
        - Implement test code
~omit~

■After

- Bucky-Core
    - Overview
    - Feature
    - Setup
        - Install
        - Implement test code
        - Set connecting information for database
    - Usage
        - Run test
        - Rerun test
~omit~

ruby version

Because forgetting specific the required_ruby_version in gemspec, the required version is showing greater than v1.0.0 at that time. We fixed that in a hurry.