Firmware testers

Updates and News
1

I just had an idea. I was sitting here in dread of updating and testing all the firmware…again.

Who would be willing to be a tester, official tester, or something? How much would an appropriate amount be to pay for this?
We have 7 major builds for the CNC’s (series and dual), plus the Zen (ramps and mini), and MP3DP (easy for me to test).

I think if we could get someone with each of the standard builds with a board and they ran the crown and maybe one other test file from an LCD we could call a firmware “verified”. It would also kinda need to be done in a timely manor, a day or two. Probably once every month or two? Just looking to make sure the LCD works as expected and the steppers move the same way. Not looking for super deep testing or anything.

I would love to be able to run updates a little more often and not dread them so much.

1 Like
2

Don’t listen to Big Psych. Regular doses of existential dread builds character! :clown_face:

3

I think it is possible to overdose.

1 Like
4

Inoculate yourself with consistent, low dosages of Lovecraft, Poe, and Fox News…

5

Also helpful: Edward Gorey…

6

In all seriousness, I would be more than willing to test builds based on my rig (RAMPS, dual endstops, A4688s, could change…), once I get it up and running. As for payment… I dunno… I’d almost feel bad taking anything for it. But if you feel the need to compensate your testers, I’d be willing to do it for the odd bit every now and again (not even every test).

7

What about a system like this (you know I love a good system):

  • A quick checklist of steps and things to check. Basically limited to crap that has bit use before (LCD issues, Z speeds…)
  • A page (dl page) with bleeding edge releases, or beta type releases
  • A page with a form to confirm that you tested it, and maybe a few questions about your build and a place to detail issues. This would be linked from the dl page.

Then people could just choose to help you out, or not, and it would be clear that this code was beta, because you are asking for testers.

When you get a couple of :+1: on a build, feel free to release it?

I guess the limit is, you need to trust that a build is good before you flash and ship a board.

The trouble with paying is that the test is simple, so it shouldn’t be too expensive, but the users are experts, and has strong skills, so they probably demand high pay rates.

You might get a little higher quality and better rest if you had the changes peer reviewed. The extra eyes can really catch a lot of fumbles.

2 Likes
8

You might limit your tester pool, but including things like “image of crown drawn with firmware” would be good. Hell, create new “testing” images to run, even if they’re just the crown with the build info on it, and have the testers run those crowns for testing and uploading. I’m learning a lot about test evidence working in a real QA team.

1 Like
9

The biggest issues are not compiling, I can test that easy enough. After that all sorts of weird little things happen, and I think will be found really easily. A list of some basic things like, clockwise on the knob moves the selection down, or somethingllike that is quick and easy, 10 steps or so. For whatever reason I miss things way more often than I should, I just don’t like doing it so I think I do minimal effort testing.

Good idea, super fast and proof of work.

It needs to be super standard, under 3’ build, drv8825 for ramps, windows, all out of the box standard as if bought from here.

10

Hrmm… 32"x32", been thinking about getting some DRVs anyway, given how cheap they are, and I might have to pick up some bare microswitches. The kit I bought has endstops, but the boards they’re attached to only provide NO connections… :angry:

11

I can help test, now that I have a rambo to flash. Also have a ramps. Both will be for the mpcnc, my lowrider is too funky to test anything.

1 Like
12

I was just hoping to find a person for each variant that has build that gets used often that can flash and do a quick test. I would hate for anyone to have to do a bunch of wire swaps and all of that.

13

Or…maybe I do. Maybe this should get hired out…a more extensive thorough testing. shoot.

14

I will be more then happy to test it. My build is a little bigger though MPCNC 24x48 Rambo dual end stop from V1… No payment needed.
I like the idea of the test file name on the Crown…

1 Like
15

(sorry, this got really long while I wasn’t watching)

Testing can be thorough and consistent without necessarily being arduous or expensive. I’d suggest building very explicit, step-by-step test procedures that can be executed by a very basic skill-level user. Remember the goal of testing is to identify defects (or ideally ensuring there aren’t any). Testing does not have to include defect troubleshooting/investigation/remediation -that’s where the expensive expertise comes in. The easier testing is to complete, the more likely you are to get complete test results.

There are probably other more expert folks available, but when I’ve documented test plans, I’ve used spreadsheets or tables that serve both as user/tester instructions and result collection tools. In a spreadsheet, I’d have a header worksheet to capture tester name and hardware variations, then a separate worksheet for each isolated test activity to be completed. The worksheets might be things like:

  1. Compile and Upload Firmware
  2. Exercise LCD UI and verify version
  3. Browse SD contents, load and run test gcode file

Each worksheet would have columns like Step #, Description of user action to be taken (e.g. “Rotate knob 5 clicks clockwise”), Expected result (“LCD scrolls down to display “Settings” menu”), Pass/Fail, and Actual Result (a free entry field where tester describes what the failure was, e.g. “LCD display scrolled up”). If the tester can workaround a single-line failure, they continue with the subsequent tests, if not they note that testing could not continue.

This makes it is easy to do consistent testing even with different users executing the tests, and to collect consistent results, including some “hints” if things don’t go as expected.

Whoever collects the test results does the final analysis, not the tester. Obviously, if all the individual steps pass, the test passes, but someone else needs to make the call as to whether a minor unexpected result requires fixing before the release.

It seems daunting to put this together, but you really only have to do it once, with minor updates as features change through the releases.

2 Likes
16

@vicious1 Don’t feel bad about this. There is a reason most development shops have a QA department. I don’t have QA people but in my workflow I require the person requesting changes to verify those changes in both a test environment as well as once the test environment has been synced with production. As they say, you can’t see the forest through the trees, meaning when you’ve been looking at code for so long you WILL MISS something, it’s almost guaranteed. Shit happens, being prepared is a good CYA move.

17

I will jump on the band wagon and say, these are not unique experiences. Which is why Continuous Integration and many other tools have been created to improve software quality. From the outside, it may look like those things solve really complicated bugs, but in reality, they just find missing semicolons.

2 Likes
18

It’s scary how true that statement is…

1 Like
19

The most evil prank I ever conceived of in college was wrapping gcc in a sed script that would remove random semicolons…

Second most evil was moving the contents of someones home directory into one of a three-deep tree of 16 directories each (basically a 3-digit hex number).

Third was cat’ing Simpsons audio clips to their sound devices.

1 Like
20

Oh man. If you ever changed gcc to be something else, I would have no idea… I trust it so much.

The home dir thing wouldn’t be hard to undo du -s | sort -n.

1 Like