[Hamara-devel] testing and QA, maybe something to think about when fund-rasing as well

[shirish]

Hi all,
First of all, I hope everybody has gone through yesterday's IRC logs so 
that everybody is on the same page. As soon as I get some more details 
of the product and share it with Vikas, somebody from the team would 
have to go and talk to them (probably a month from now) . As you are 
geographically so much closer situated to them it would make more sense 
for somebody on the team to talk to them rather than me. It doesn't make 
sense for me to come all the way over there when you guys can easily 
talk to them.

While I was talking to him it did struck me that we haven't talked about 
QA and testing at all apart from some passing references by Aparna about 
using testopia we haven't really discussed anything.

In the GNU/Linux world there are all kinds of tests that you could do 
for an application, library or whatever.

For instance, there is this :-

https://github.com/linux-test-project/ltp

This is more towards the kernel rather than the desktop but the idea 
remains the same. There is lot to be gained there as in expertise but 
also quite a bit of investment as in time.

I am sure people are aware of tools like gdb where people can use to get 
stacktraces when crashes happen but also use them to step through a 
program using breakpoints. There are similar tools for almost all the 
languages and bindings which perform similar task.

There is also valgrind where you can see if the memory is leaking 
somewhere but again needs lot of time, energy and of course a somewhat 
beefier system so that you can use the program and see if there is a 
memory leak, report it. Again lot of time and patience is needed.

There is also the non-free phoronix-test-suite which also does similar 
activity as well


─[$] aptitude show phoronix-test-suite
 

Package: phoronix-test-suite
State: installed
Automatically installed: no
Version: 5.2.1-1
Priority: optional
Section: non-free/utils
Maintainer: Debian QA Group <packages at qa.debian.org>
Architecture: all
Uncompressed Size: 2,637 k
Depends: php5-cli, php5-gd
Description: comprehensive testing and benchmarking platform
  The Phoronix Test Suite is the most comprehensive testing and 
benchmarking platform available for the Linux operating system. This 
software is designed to effectively carry out both qualitative and 
quantitative benchmarks in a clean, reproducible, and easy-to-use 
manner. The Phoronix Test Suite consists of a lightweight processing 
core (pts-core) with each benchmark consisting of an XML-based profile 
with related resource scripts. The process from the benchmark 
installation, to the actual benchmarking, to the parsing of important 
hardware and software components is heavily automated and completely 
repeatable, asking users only for confirmation of actions.
Homepage: http://www.phoronix-test-suite.com/

It again needs a somewhat beefier machine, the only downer is the 
results are limited to phoronix.com

I am sure people might have heard of black-box testing where you have 
sort of random tests happening on the product or system, both the input 
and output would be fuzzy but at times pretty useful.

Such kind of tools have also come to the FOSS world as well. See for e.g. :-

[$] aptitude show fuzz
 

Package: fuzz
State: not installed
Version: 0.6-14
Priority: optional
Section: devel
Maintainer: Jari Aalto <jari.aalto at cante.net>
Architecture: amd64
Uncompressed Size: 82.9 k
Depends: libc6 (>= 2.12)
Description: stress-test programs by giving them random input
  The fuzz generator is designed to attack certain kinds of software and 
expose one particular kind of bug common in software. This is the 
situation where the programmer implicitly makes some assumptions about 
the data stream that the program will be parsing. If the data stream is 
substantially different then the program might not be able to deal with it.
Homepage: http://freshmeat.net/projects/fuzz

I will share an example of a tool I use from time to time :-

[$] dpkg -l youtube-dl
 

Desired=Unknown/Install/Remove/Purge/Hold
| 
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version              Architecture 
      Description
+++-==============================-====================-====================-=================================================================
ii  youtube-dl                     2015.06.04.1-1       all 
      downloader of videos from YouTube and other sites

It's a command-line tool to download videos or/and audio from youtube 
and others.

I use it to download videos as just like wget if it gets disconnected I 
can resume it.

Anyways, I use perhaps 5-10% of the different options the application 
can do. Now while that part gets tested there are many within which 
might bit-rot, become stale or something. For e.g. there are something 
like 60-70 sites from which youtube-dl can download and about 10 odd 
ways in which the audio and video can download. You can see the 
permutations and combinations that are and can be there.

With a bit of luck, the tool may find if something is broken. Again 
there would be significant time investment for learning, understanding, 
testing and seeing if something comes out of it.

Just like documentation, testing and QA have been neglected because of 
many reasons :-

a. It is boring, not lot of "action"
b. Requires significant time without anything to show
c. Have to also be able to distinguish false positives and tell upstream 
about those so it's fixed in the next version/s.

There are many many more tools which look at the problem/issue from 
various angles.

I do hope that if and when we do fund-raising we also think of testing 
and QA, budget for it as well as talk about it in fund-raising as well.

And I haven't even got into auditing which is a different ball-game 
altogether.

Once we have the alpha, these tools can come into play and people can 
use it to see what works, what breaks and generally have a feel for tools.

Look forward to feedback.
-- 
Regards,
Shirish Agarwal,
Community Lead,
Hamaralinux.org