Git is powerful... and very painful the first time! / Github contribution howto / Tweaking Marlin for better 3D printer menus.
 |
| Git amazing mess, only partially resumed by Oliver Steele. Indeed, the upstream higher repository is not shown here, as seen on this other bigger, clean and useful cheat sheet. |
I needed it for my Delta printer: these printer do require precise calibration, often with a sequence of gcode commands (check the end of this post for more and why I wanted these to be in my menus and not as initialization files on all my SD cards nor on a PC over a USB cable).
Now, Marlin is hosted on github, a community front end to many other open source projects. Actually, the linux kernel itself is developed with git so it works, for sure.
I often tried to use github, but I never went past a simple git clone of a repository. It was still better than to download a zip archive, because you can easily get the new stuff with a git pull. But here and partially out of curiosity, I wanted to try and contribute to a project at the source.
Now... what a huge and painful procedure just to give a hand! Seriously, it is mind boggling how much crap and megabytes need to be handled just to help and submit a few dozen f*king lines of code to an opensource project hosted on github. This is too bad since I am sure many programmers would be glad to give a hand to project they stumble upon (like me, often), but without this need to become administrators of complex and shared projects themselves!
This article is all about posting and contributing to a project hosted on github. You'll get the calibration g-code line I used at the end of the post if you really ask yourself.
So the deal is that git requires you to make yourself a full copy of the project, both remotely and locally. Then you work locally, you commit locally, then again remotely, and then tell the project maintainer that your copy has a change that may be interesting to them. Powerful, but convoluted for sure as it also means you need to get the changes from their own project that also moves on in between.
Anyway git and github are painful enough for me to record the full procedure here, and hopefully once for all. It may help some old school developers as me, and may be it would let them contribute more easily to open source projects hosted on github.
Initial setup: create your own version of the original repository
OK. For now you want to use your browser at https://github.com/1) You first need (to create) a github account (not that much painful so far, but it already gets annoying haha)
2) Then go to the github repository you want to work with.
For Marlin, the main one seems to be https://github.com/MarlinFirmware/Marlin.git
You will see that there are tons of secondary variants made by tons of people so make sure you are working on the "right" one. Now, this is really annoying to me when your goal as here is just to try to submit a piece of code to the "original" authors. And no, you have to create your own repository first!
This just due to the way github works: as a matter of fact we have to create yet another version of Marlin!
3) Now, still in the web page, click and fork the official Marlin repository (it will create copy in your account... and add to the mess for newcomers!).
4) Make sure to rename this copy to a meaningful name. Be it for you or for others, it will prove very valuable! Just click on the tiny almost hidden "project settings" icon in the bottom right margin (see below), then rename your project.
Now get back to your main project page by clicking on the name of the repository, then copy the content of the small box in the bottom right margin, labeled "HTTPS clone URL". This is your repository URL, its ID card in a way.
5) Now back to the console!
WARNING! You may want to use your public SSH keys to log in (so that you are safe and do not have to provide your login all the time). In this case, do NOT use the above link, but tweak it to this syntax instead:
git clone git@github.com:MoonCactus/Marlin-JFR-features.git
Else, to use other means to identify yourself on github, you may use the copied URL as is:
git clone https://github.com/MoonCactus/Marlin-JFR-features.git
Of course you want to specify your own repository, not the one you forked from.
Also, make sure you are using https and not http or will will make your life harder later on.
Now you can code... (but the pain is still to come!)
Now the funniest part: start coding and hacking your stuff around... Oh you did already, didn't you?And you though about committing your changes to git afterwards, right?
In fact, I use a comparison tool like meld to re-incorporate my changes to a forked repository after I usually start working "just to try" on a cloned repository. There should be one single command to switch from the latter to the former... Alas, do not forget: Git was first, and only then came the source code... ;)
Now, it is time to submit your changes, and here it becomes annoying again.
Commit your changes... multiple times and all everywhere ;)
When you are done and want to save/store/submit/commit your changes, you have different commands to issue (no way it will never be as dead simple as a dumb cvs commit!).- git status -- a useful command to check the state of your repository
E.g. here it tells me I modified three files in the project:
On branch Development
Your branch is up-to-date with 'origin/Development'.
Changes not staged for commit:
(use "git add
(use "git checkout --
modified: Marlin/Marlin.h
modified: Marlin/Marlin_main.cpp
modified: Marlin/ultralcd.cpp
no changes added to commit (use "git add" and/or "git commit -a")
- git add -- Tell it you changed something (!)
You first need to commit your changes locally with a git add followed by the name of files that changed (listed above):
cd Marlin
git add Marlin.h Marlin_main.cpp ultralcd.cpp
- git commit -- Now tell it to record that you changed something, still locally (!)
In fact you may merge the add + commit commands by typing git commit -a instead.
But no, it will not be uploaded yet. It may be useful though, so that you can work with a local history and you can work offline. But that is nonetheless more steps to the whole procedure (what the heck is then the need for an index and a git add really?). Anyway here it goes:
git commit
[Development dd301be] Added suport for multiline G-code commands in the
LCD menus
3 files changed, 43 insertions(+), 4 deletions(-)
As you seen it asked for a message related to the changes. Better write something useful because it will show in the history, and it helps you reverting to an older revision of the project when you screw it all, and it helps anyway to understand what you did months ago!
- git push -- At last, send it to your repository!
Hooray? oh no, not at all!
First, it will ask for some more obscure options. Better do the following once before you push your changes:
git config --global push.default matching
Then again:
git push
But now it asks for your github credentials. Here I typed them manually:
Username for 'https://github.com': MoonCactus
Password for 'https://MoonCactus@github.com': XXXXX
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.22 KiB | 0 bytes/s, done.
Total 6 (delta 5), reused 0 (delta 0)
To https://github.com/MoonCactus/Marlin-JFR-features.git
865ca0e..dd301be Development -> Development
At least it worked! My remote repository on github is now updated with my local changes. But how do I get the remote changes from the original repository, and further, how do I tell them I added a feature to Marlin?
Credentials and github
Don't we just want to submit one patch and not to administrate repositories? No way...
As we have seen, git push asks for your credentials (login and password). And it is more likely we will be making changes and submitting them more than once.
To avoid having to type it all the time, you may check the many options given in this post.
If you are using a desktop PC and use an encrypted user directory, then you may take the risk to store your password in your ~/.netrc special file (I have no clue on windows, sorry there!).
But you would better go the safe way and upload your public SSH keys to github. This way it stops asking and you stay "fully" secure.
You can do that here on the web:
github / edit profile / SSH keys (aka https://github.com/settings/ssh)
There you can add your own "*.pub" SSH key. Check this help if you need more about SSH keys, or if you want github to generate some for you because you do not have some already (this is not recommended imho).
Then, check you are OK when ssh -T git@github.com no more asks for your login.
But still... for SSH login to work, you must have cloned your project NOT with the given https repository link, but the following way instead. Else you are screwed and need to download the project all over again, by using:
git clone git@github.com:MoonCactus/Marlin-JFR-features.git
Did I say git was just plain painful to set for newcomers? Well, at least you can first commit your existing changes in the "htpps style" by providing your login to the git push, then delete the local project and re-clone the project as just above.
So once done you can check your local and remote projects are synced:
git status
On branch Development
Your branch is up-to-date with 'origin/Development'.
nothing to commit, working directory clean
And git push should no more ask your ccredentials
git push
Everything up-to-date
Getting remote changes down to your local version and "syncing a fork" (sounds weird, eh?)
It is easy to get the changes from your repository to another PC. First you must clone the repository the same way you did above to your other PC. Then when you want to download the changes that were pushed from elsewhere, just do this:
git pull
But... this only gets the changes from your own repository, and not from the initial upstream repository. As usual git does not make it easy. Unless it is a one-time help you are giving to a project and you are OK to redo all the mess above months later, syncing with upstream is quite important. Otherwise and with time, your project will drift away from the "official" build to the point it will be much harder to get and merge the changes while dealing with increasing discrepancies and even conflicts. Submitting patches to the original authors will be even more difficult.
You first need to tell that your version should also look for changes from within the original project you forked from (the upstream).
OK let us try to finalize our stuff so that it will also get the changes from the upstream.
Get its original URL from github, and add it to the local copy this way:
git remote add upstream https://github.com/MarlinFirmware/Marlin.git
Now we check all the repositories which changes will be taken from:
git remote -v
origin git@github.com:MoonCactus/Marlin-JFR-features.git (fetch)
origin git@github.com:MoonCactus/Marlin-JFR-features.git (push)
upstream https://github.com/MarlinFirmware/Marlin.git (fetch)
upstream https://github.com/MarlinFirmware/Marlin.git (push)
Now, to download specifically the changes that are made in the original repository into your repository, we need more obscure and brain-damaging commands:
git fetch upstream
From https://github.com/MarlinFirmware/Marlin
* [new branch] Development -> upstream/Development
* [new branch] revert-1350-Development -> upstream/revert-1350-Development
Branches? Only monkeys needs branches!
 |
| Or... a typical hellish source code sharing system! From this answer at stackoverflow. |
Branches allow a project to move on while a stable version of the sources is kept. This is mostly for safety and to "freeze" the state of a project in a "stable" state, where only bug fixes are made. New features are made only in a development or experimental branch. Then you "simply" switch between the development and production branches, according to the current work that need to be done (do not get confused!). Indeed, when you want to make it funnier, you can create many branches, give them obscure names and get changes from one withing another and so... But this is another hellish topic, i.e. advanced uses of git that I do not need here!
Our own sub-project may not need branches... But the original project does have a few, and we thus need to deal with them. In our case, the second branch named revert-1350-Development just looks enough dubious to me so I will not even go and try to get more information about it... I will assume the first is the only useful one. Surprisingly there is no "stable" or "production" branch for this project!
git merge upstream/Development
When also dealing with local branches and not-yet committed changes, you will need to read about more gory details (drowning a spoon, oh no, syncing a fork).
More useful imho is this comprehensive, dynamic, and well done cheat cheat.
More useful imho is this comprehensive, dynamic, and well done cheat cheat.
Pull requests, aka submitting a change to the original authors... at last!
Everyone benefits from a project that is maintained in a good shape. This is also why there are often separate "stable" (even sometimes obsolete), and development branches. The stable branch ought to be fully functional, with only bug fixes pushed into it.
So, make sure your own stuff works well! Do not submit broken stuff to programmers or they will hate you :)
Also, make sure your new feature or bug fix is useful to others, and that it neither breaks something far far away in the original code source... or they will hate you alike.
You need also to comment it properly and follow the guidelines (code style and so). Anyway, your patch will probably be refused if it does not seem good enough for the maintainers! And they may also hate you ;)
So enough joking... the deal is to tell the maintainer that your branch has something new, that they can grab and integrate it in their own (upstream) repository. This is mostly why your changes should be minimal: be as close as possible to their latest versions before you submit your so-called pull request.
Strangely, the pull request must be made from a web browser, not from the console.
Actually here the official documentation is linear and easy to follow. May be this is because it moves a bit away from the overly complex guts of git into something much easier now (and I really am a console guy though!).
So the next steps is to ask the maintainers to integrate your change. And this is made straightforward in the official documentation this time.
> Still, feel free to ask when you want me to explain it further in this post !
I just hope this post could help a few developers to contribute to open source projects.
 |
| A sierpinsky triangle... made with git by BooK ! |
Addendum: add your own LCD controller menus in Marlin
Just clone the source code from https://github.com/MarlinFirmware/Marlin.git , then open ultralcd.cpp and to the function named lcd_prepare_menu() for example.
There you will see commands like this:
MENU_ITEM(gcode, MSG_DISABLE_STEPPERS, PSTR("M84"));
This just adds a command named thanks to MSG_DISABLE_STEPPERS (check the translation strings in the project files!), and that inserts the M84 command in the current buffer that is being processed.
But the existing state of Marlin will not allow more than one command.
My Delta printer used M7XX commands to calibrate the level of its bed. These functions have since disappeared from Marlin (replaced by something else). But the full story was:
M702 ; reset the leveling bed values
G28 ; home
G1 X-77.94 Y-45 Z36 F8000 ; move above first corner
G4 S3 ; wait for 3 seconds
M701 P0 ; record first calibration point with the Z probe
G1 X77.94 Y-45 Z36 ; go to second calibration point
G4 S3 ; and so on...
M701 P1
G1 X0 Y90 Z36
G4 S3
M701 P2
M700 ; compute and store the leveling values
G1 X0 Y0 Z100 F8000
This was definitely not one unique g-code command... With the patch I have just submitted I now write it this way:
MENU_ITEM(gcode, "Calibrate bed", PSTR("M702\nG28\nG1 X-77.94 Y-45 Z36 F8000\nG4 S3\nM701 P0\nG1 X77.94 Y-45 Z36\nG4 S3\nM701 P1\nG1 X0 Y90 Z36\nG4 S3\nM701 P2\nM700\nG1 X0 Y0 Z100 F8000"));
I also added a "hovering/spiral" triangular movement above all three corners to check that my bed is level. First pass is 1mm above bed, second pass is 0.3mm above bed, and slower:
MENU_ITEM(gcode, "Check level", PSTR("G28\nG1 X0 Y0 Z1 F4000\nG1 X-77.94 Y-45 Z1\nG1 X77.94 Y-45\nG1 X0 Y90\nG1 X-77.94 Y-45\nG4 S2\nG1 X-77.94 Y-45 Z0.3 F2000\nG1 X-77.94 Y-45\nG1 X77.94 Y-45\nG1 X0 Y90\nG1 X-77.94 Y-45\nG1 X0 Y0 Z0"));
Finally three more commands, to insert or retract the filament:
MENU_ITEM(gcode, "Retract filament", PSTR("M302\nM82\nG92 E0\nG1 F4000 E-800"));
MENU_ITEM(gcode, "Insert filament", PSTR("M302\nM82\nG92 E0\nG1 F4000 E60"));
MENU_ITEM(gcode, "Finalize filament", PSTR("G1 F4000 E790"));
Once done, you must run the Arduino IDE, open your modified Marlin project, rebuild and upload the firmware to your printer. And here it is, a nice and tailored firmware menu entry:
 |
| Improved menu entries in Marlin with multiple g-code commands. This is very useful to calibrate the bed of my Delta printer. |
Without the patch to Marlin, I had to save these to separate files on each of the SD cards I could use with the printer, or to send them over USB to the printer from a PC, which I do not like at all. Another solution would have been to write a dedicated command in ultralcd.c and attach it to a menu entry.
What about Printrun custom buttons? Multi line commands support alike!
By the way, I used Pronterface a lot during the initial calibration of my printer. It has very convenient "custom buttons" but they suffer from the same limitation: you can only give them one g-code command!
In fact, copy/pasting a sequence of multiline commands does work, until you restart restart Pronterface since it then keeps only the first line. This was quite annoying because of the lng sequence I used to calibrate my printer.
It can be tweaked very easily, by adding the line in bold to the following function in printrun/pronsole.py:
def precmd(self, line):
line= line.replace("|", "\n") # JFR allow multiline GCODE button
if line.upper().startswith("M114"):
self.userm114 += 1
(...)
This way, custom buttons may hold "multiline" commands, by separating them with a "|" (pipe) in place of a newline "\n". This tricks lets Pronterface reload the buttons with no additional fix to its source code.
update: oops. The author just told me I could instead have used the "macro" option, and then have added a custom button that plays the macro. This also does keep the newlines unchanged. Still, it is a bit counter-intuitive that the custom button shall be restricted to a single command imho.
update: oops. The author just told me I could instead have used the "macro" option, and then have added a custom button that plays the macro. This also does keep the newlines unchanged. Still, it is a bit counter-intuitive that the custom button shall be restricted to a single command imho.
And no, I will certainly NOT set up an entire github fork just for one single line of source code! :D
 |
| This is a modified Pronterface that allows multiline gcode commands in the custom buttons! |