I’m trying to consolidate the bits and pieces around release process ideas discussed in OMG!!!!! Extruders = 0 and Marlin 2.0! under a separate topic.
TL;DR; - With Marlin 2.0 released and EXTRUDERS = 0 setting now mostly functional, albeit not officially supported (@jeffeb3 was able to squeeze a test to upstream that should ensure it at least compiles), it’s time to take a deep look into how V1 firmware releases can be more automated and streamlined.
Config file management experiments.
I used four different methods to upgrade the configs in allted/Marlin (as of Oct 23 2019 changes) to match the latest in Marlin 2.0.0. Please note that the branches I’ve posted here are a faithful representation of old configs e.g. I did not change configs to Version.h nor EXTRUDERS 0 yet nor performed any cleanup and unification. There will be a follow-up 
Method 1
config/default (stored in allted repository).Method 2
config/default from the same point in time (stored in allted repository).I’m too lazy to do this kind of work by hand so it was scripted
Method 3
Method 4
Method #3 is very similar to what @vicious1 has been doing so far, the only difference is that I applied the diff to a set of “example” configs using git apply instead of creating a separate branch for each.
Results
Methods 1 and 2 yielded similar results: I saw conflicts in exactly the same files, made a few mistakes when resolving the conflicts in Archim2_T8_16T_LCD_16step on both occasions. Overall the experience was mostly identical and I would call the chance a human error the same irregardless of in which “direction” patches are applied.
Method 3 had conflicts in fewer files however it did result in more discrepancies, some of these in files that did not report a conflict.
Method 4 generated correct configs as expected (no human error), however it has it’s deficiencies:
However the most annoying issue is that opt_xxx commands are “blind” and will not complain if they can’t find the variable to be set/enabled/disabled. This may be a problem because automation will fail “silently” if upstream renames a config value.
So overall it seems that although config generation can be scripted, it may be better to automate the patching of configs instead. Sure, this may require more manual intervention when patches fail to apply, but it has a lower chance of silent errors and the generated configs do not lose valuable information stored in comments.
Take a look at the diffs I posted, and let me know what you think.
I’m very tempted to say #4 is the way we should go. I would even say we could go a step farther and say that we could make a config for CNC or 3DP and apply those in series with a MRambo config. They certainly seem much easier to maintain and understand than the patch files.
We could rewrite the opt_enable functions to find and then replace/check. There would still be an element of dumb scripts editing smart code. But hopefully other mistakes would end up with more catastrophic errors (so they would be easy to detect, like compile errors).
It seems like your script could reverse the process too, right? So if Ryan was having trouble, he could go edit a Config.h file, get it working, and then guarantee he could get everything he changed back into the configs? Keep in mind, his git education is still in it’s early stages.
Finally, the other question I have is what is the outcome of this. Users need to have a .zip, with the configuration for their setup, at least for the supported set of configurations. We also need to be able to support getting upstream updates, and I know you have ideas of making a CI builder make nightlies, or other bleeding edge builds.
a) Do we keep the branches for each version?
b) Do we make branches follow their upstream values, and keep our configs in examples? If so, do we have a script to build the .zips for release?
c) Same as b, but keep our configs just in the raw form, and build the .zips from the configs?
Well there is a butt-ugly awk script that I used to generate the scripts for #4. I’m not sure how maintainable this would be in the long haul though. I would not bet on that …
I’m really torn on #4. On one hand I like the fact that it’s a really compact and clean representation of differences from a “base” and the loss of comments, although annoying, can be minimized if we can get a “base” MPCNC config included in Marlin upstream. They do maintain their “example” configs and there aren’t that many CNC ones yet. I don’t expect them to take all of V1 configs but a generic MPCNC one could be accepted, I think it’s worth a shot. This way all comments about JD and arc length etc. will remain intact because we only script the deviations from the base MPCNC config that just changes a board type, currents, stepper drivers etc.
The “dumbness” of the tools is a risk though. We can attempt to improve these, however sed has trouble counting the number of substitutions. I found some ideas from a related post but it’s not clear if this can be done. If not, we’d have to convince upstream to accept a rewrite of these to awk/perl. I don’t know how open they would be to a change like that. We can sure try.
I’m still trying to make up my mind here 
Github has workflows now so we can do all kinds of things even outside of CI. I explored actions yesterday and there are tons of stuff. One can check out multiple repos/branches, execute shell commands, create pull requests from scripted trees or just commit these etc. I have to play around with this stuff a bit to see if it works the way it’s advertised.
My preference would be to store all V1 configs under config/examples directory and release a single source ZIP for all machines. Or maybe even just tag a release and point people to GitHub’s zip download.
For automated builds, I’d like to make pre-built firmware files available. I was digging around a few days back and found firmware updater that plugs into Octoprint. The nice thing about this is that it can take a HEX file or even a URL to a hex file and should support both ATMega boards and SAM boards (such as Archims). Not sure how easy/hard it would be to include that and all the tools it needs to do the flashing in V1Pi .
Here is a fix for sed. but also, we can easy build in a grep first. Then we can just set -x or set -e (I always forget which is which).
Interesting. There may be light at the end of that tunnel after all. They do look for GNU sed in these scripts so “q” being a GNU extension should not be a problem …
On another note. I got this reply from an engineer at Ultimachine about Archim boards and SD-card update.
We have not tried to program over SD Card. It is possible if a bootloader program is made and put on the device ahead of time according to the datasheet.
So in human-readable terms, they haven’t done the work to find/build a bootloader to support this.
Every step to make flashing the arduinos easier is important. Even copying from examples/ to Marlin/ is enough to trip up a lot of people. There are all skill levels here, including people who are really good with wood, but don’t have a computer near their shop.
I have used this, and I don’t think it would be hard to include in v1pi, but I am guessing there are maybe 1% of users with a pi. It is a nice feature to add, but certainly not the default path. I feel similarly with the 32b boards. They may be the future, but are certainly not mainstream for v1 (yet).
We could probably configure this to create .zips of each configuration from a branch and keep them aside. Then when a build becomes “stable”, Ryan could manually publish those .zips as a github release.
I’m a little worried that you and I will create this great system, and Ryan won’t be able to use it. Having github build .zips for him to test would probably be the easiest way to ensure he doesn’t have to fight windows with our scripts.
I am not sure that compiling Marlin is a better default path. There has to be a way to allow less technical folks to update firmware without jumping through gazillion hoops that this involves. If it calls for getting an additional $35 worth of hardware then why not sell V1Pi-s in the store
What we need, is a program like etcher that works in windows and flashes the .hex. That would probably be enough.
Looks like Scott is also looking at github actions:
Well there is this thing but it doesn’t look like a very active project. Then again avrdude hasn’t changed for years. For RAMBO boards, maybe PrusaSlicer will do the trick. I haven’t tried to use it to flash a HEX file that is not from Prusa though. Maybe they have some “safeguards” in there.
I didn’t think the arduino bootloader used avrdude. I’m also not sure if the rambos have their own bootloader. The board manager can hide a lot of sins.
I flash Prusa firmware with avrdude so it certainly can do it:
avrdude -v -patmega2560 -cstk500v2 -P/dev/ttyACM3 -b115200 -D -Uflash:w:../Prusa-Firmware-build/Firmware.ino.hex:i
I use firmware updater in octoprint on v1pi (had to install avrdude separately). So I suppose im in the 1%. 
This is for RAMPS-like boards.
Prusa slicer would be an interesting solution.
At any rate, I think we are close.
We can have the configs for all supported configurations in one branch (or many branches for versions like bugfix, 2.0, 2.1). We will have some ci be able to configure for each setup and generate something useful for each type (.zip, .hex/.bin, or both).
This can be done on almost any branch, and when we get one that is “stable” we can make a release, tag that one branch, and upload the build products to the release page.
I think you might be the 1% of the 1% 
Given I’ve never been a big fan of Arduino IDE, I had to google this 
It does seem to include avrdude. There is a post that explains how to “harvest” flashing command from the console The simplest way to upload a compiled .hex file without the IDE - IDE 1.x - Arduino Forum