Here you go ... #10
Replies: 137 comments
-
|
Nice, that's wild! I will try to look through it and see how did you solve those missing ones. I see you like more "just make it work" than I did! |
Beta Was this translation helpful? Give feedback.
-
|
Well, it took me quite a few days to get it all running, knowing absolutely nothing about Jenkins or how VyOS is built. So that was a steep learning curve and I just wanted to get things done, because I was running out of time. |
Beta Was this translation helpful? Give feedback.
-
|
Well they took some packages from the Nvidia networking operating system cumulus, so that is one thing. There is also one package repository in which they don't have the |
Beta Was this translation helpful? Give feedback.
-
|
Yea, same for me, I didn't know Jenkins week before. I was just using docker to build ISOs - no knowledge about the packages or build system, so a lot learning for me as well. You did make run like everything, that's crazy! I don't even think about dealing with the "current" or docker-vyos or the vm stuff. Also it's crazy that vyos requires so many patches to make work! Looks like they just hope they don't need to rebuild what they have I guess since 1/3 is broken... |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I'm quite happy it now builds itself and I just have to rebase it :) Hm, there are some things where I'm unsure whether it is missing knowledge or if those things were left out on-purpose. For example the go environment variables that they write to Go to https://blog.flowblok.id.au/2013-02/shell-startup-scripts.html and scroll to the image under Implementation, then figure out how Jenkins messes with things and how they built their docker build-vm. That is a nightmare. EDIT: I ended up just removing that check in the docker build process in |
Beta Was this translation helpful? Give feedback.
-
|
Here is my list of Jenkins projects with the branches that I successfully built. I named the packages in hvinfo Make sure you create Views in Jenkins, by default only the "All" view exists, otherwise keeping track of it all is a real problem:
|
Beta Was this translation helpful? Give feedback.
-
|
Well at least we are having fun ;) |
Beta Was this translation helpful? Give feedback.
-
Nah, that is normal it seems. The problem is that they wrote it to the one file that is hard to source non-interactively. Go look at that nice image in the link I provided. |
Beta Was this translation helpful? Give feedback.
-
VyOS Package RepositorySee https://wiki.debian.org/DebianRepository/SetupWithReprepro. Jenkins -> Manage -> Credentials -> Global -> Add Credentials:
Jenkins -> Manage -> Configure -> Global Properties -> Environment Variables -> Add: (depending on whether you are using DNS or not)
Add nginx site: @dd010101: See above for |
Beta Was this translation helpful? Give feedback.
-
|
@dd010101 may i ask "what" was just deleted? i'd like to know for context ... |
Beta Was this translation helpful? Give feedback.
-
|
Before I forget, don't think about building by git tags via Jenkins or another way. That leads to problems because the built packages will be put into a repository named according to the git tag, i.e. you get the repository 1.3.7, while most of your packages are in equuleus. That leads to lots of problems. Just see that you build it all when they stop making changes in the LTS repositories. Best chance we have of getting a nice image that is as close to the original as possible. |
Beta Was this translation helpful? Give feedback.
-
|
Just a note, debian removed the backports for the buster debian version, even from their archive server, somewhere end of April iirc. As a result I had to build some more dependencies sometimes for the equuleus branch, as they weren't any longer available. |
Beta Was this translation helpful? Give feedback.
-
|
I found this as well with vyos-xe-guest-utilities where they built sagitta package from current branch but their script will try to put it in current... So this is my solution - https://github.com/dd010101/vyos-jenkins?tab=readme-ov-file#uncron :D There is no way to get replicate of their release since the versions of vyos are basically timestamp of the build. One would need to build the ISO in the time when they do otherwise you have version that is ahead by the time you get to built it from the time they built it. As there is no global tag that would say 1.3.7 has these versions of these repositories, the build gets whats available at that time... So there is not point to make build to match the build of vyos, it's simply random... |
Beta Was this translation helpful? Give feedback.
-
|
I mean sure, that is one way to do it :) As I'm using uncron because of concurrency problems otherwise, because my Jenkins build VM is set to 8 concurrent tasks, that wasn't an option. But of course there are other ways. I'll just hope the number of necessary workarounds will be reduced with time ... Doesn't this mean you can't build current as it is a hardcoded symlink to sagitta ... ? ;) |
Beta Was this translation helpful? Give feedback.
-
I didn't find any problems when building equuleus tho... Like the ISO build works fine with debian apt.
Well you can use the power of telepathy to sync your start of build with theirs and yes then you wound get 1:1 copy. but otherwise you don't know what versions they use since there are no tags, just commits.
Oh well... I didn't plan to support current since you can build current anyway the official way. So it's only the stable branches. |
Beta Was this translation helpful? Give feedback.
-
Regarding the result, the files that were added are from the vyos git repo, correct? Is it possible that they are from the previous version that was initially commited and potentially go unused, because we use the debian folder to build the package? |
Beta Was this translation helpful? Give feedback.
-
I'm afraid I don't know how to answer this one. Maybe this helps? and from upstream tarball: seems that the diff is just that Ah, and this is likely simply an artifact of me using my script to check it out which hits the build process. Yep, confirmed. |
Beta Was this translation helpful? Give feedback.
-
|
If I compare it and remove So except for the EDIT: In consequence my Jenkinsfile and build therefore could be identical and "fine", correct? |
Beta Was this translation helpful? Give feedback.
-
|
I don't see an I was able to confirm that it was my script entering the build process that added that file. Yes, your Jenkins script is effectively right. But let's be honest: the package is deprecated. There's a near 0 chance that it changes, but non-0. |
Beta Was this translation helpful? Give feedback.
-
|
@veduco Thanks for pointing the repos out and helping compare them!
I mean I'm not totally opposed, but at the same time they haven't changed those packages in a while (added by Cumulus Linux as deb package to their repo at May 18 2022; added to vyos sagitta repo with date 2023-09-25 18:36:21). There are lots of newer versions available from Cumulus Linux iirc. EDIT: Also, what is to be done about the vyos git repos without proper branches? Manually checkout a specific commit and then manually update it every once in a while? Because we can't yet be sure that newer versions added there are actually for sagitta ... EDIT2: So perhaps just wait a bit and let them - hopefully - clean it up then change it. Acceptable? |
Beta Was this translation helpful? Give feedback.
-
Doesn't matter - if you rewrite master to sagitta or commit to sagitta after a while, basically the same thing. Because if they make changes then they would want to fix the branch issue and then you have correct branch to switch to. It's good practice not to rely on totally unpinned sources, thus I did use commits for now: https://github.com/dd010101/vyos-missing/blob/sagitta/packages/libnss-tacplus/build.sh |
Beta Was this translation helpful? Give feedback.
-
|
Sure, until they have sorted the branch issues, then I propose we just follow the branch, like we do for all other vyos packages ... Less work for us. I don't think they introduce as many breaking changes (or "wrong" versions in this case) as happens with their main repos. |
Beta Was this translation helpful? Give feedback.
-
|
@veduco |
Beta Was this translation helpful? Give feedback.
-
|
I'm just an enthusiast that was irritated to not be able to update a piece of software at the heart of my network. As interesting and meaningful as the smoketests may be, I'm not set to run those as in the 2+ years of building images, I never have, and with 1.4 approaching actual GA and moving to LTS, I'd expect the number of changes to drop as well as the time between meaningful changes in builds to increase. It's a risk I'm willing to accept given the HA environment I have. I just didn't want to learn Jenkins at the time and having it scripted allows me quick access to do stuff on the side. edit: actually, thinking.. how do you guys handle custom build flavors? |
Beta Was this translation helpful? Give feedback.
-
|
I see so the build action does fix the potentional issues before build where we're forced to fork? I guess you wouldn't even have many of the issues since they were related to reprepro and other not build related steps. You would still need to have some maintenance in terms of the build actions isn't that right? Since the Jenkinsfiles contain small but some part of the build. The build commands may change as well as pinned commits. This was mine motivation to learn Jenkins since I didn't have idea of the scope, how complicated the Jenkinsfiles were but now I see most of them are simple. I don't think anything would stop you from running smoketests, you just don't see the motivation to do so? I have good experience with the stability of vyos builds without smoketests as well but I do wait fair bit before I jump on next stable just because the router is the last thing I want to debug. You are using redundant routers with different vyos versions as your HA? I'm sure going without Jenkins has its own benefits in terms that you can do what you want not what vyos team uses so it's completly valid way if you know how to build packages and obviously you don't have issue with that. I see Jenkins as good ready made solution that will handle everything for you once you have it going. If you can manage (or want or enjoy) to make your own system then perhaps Jenkins would only stay in your way. I didn't use those new build flavors yet. I'm injecting debian live hooks into vyos-build to modify the image as I see fit before building ISO, this was the only way with crux and equuleus and it works with sagitta as well so I didn't see the need to investigate the toml flavors. |
Beta Was this translation helpful? Give feedback.
-
|
Depending on the amount of modifications you are doing and the, perhaps somewhat limited, times packages change for a LTS release, your Github Actions sound like a nice solution. I'm not really worried about Jenkins though.
When my server is on I'm doing other tasks on it as well, which take longer than 2-3 hours, so running a few iso builds and smoketests all by pressing three start buttons in Jenkins isn't an issue at all. Most of the package definitions, i.e. the Jenkinsfile with the build instructions inside, I never needed to touch, meaning I have no work to do when they update them. For them all the necessary modifications are in one library function/file in vyos-build. All in all I see far less maintenance overhead as if I were to use any other method than their CI solution, which they maintain. If I want to do modifications, I can just create a branch like sagitta-mod1 in my repo and make whatever changes I like to the build scripts including adding packages, etc. It will then only require the same rebase from the vyos repo as the main branches do. The last few I did in the last week were no problem at all. Git rebase also automatically detected when they removed the patches for frr, so my commits that did the same were just skipped while rebasing, nothing manual required at all.
Well, we aren't really modifying the build process, because we only interfere with the code that stores the files/images and everything else remains untouched. That means we are using the exact same build process as VyOS is - of course at the moment theirs is quite broken so we had to fix it. Therefore we know for sure that if something is wrong, it isn't our fault - unless of course one of the missing packages we added gives us problems, but I believe the chances aren't that great. So if the smoketest fails, which just runs automatically as part of the process of building the iso, I know to just try again later. No problem for me. Or, as Jenkins stores the build artifacts, you can just look at the history of builds for your branch and pick the iso image of a certain build and download it from Jenkins. So if the smoketest fails and the iso is not uploaded via ssh (in my case) to the snapshot directory I created for it, then it is not lost, I just have to manually retrieve it. For all the ones in my snapshot folder I know that the smoketest succeeded. Just wanted to explain my thought process behind my decision. |
Beta Was this translation helpful? Give feedback.
-
|
We could even add a Jenkins parameter for the iso build process to add modifications or packages. The packages one is easy. From the The documentation for it is here https://www.jenkins.io/doc/book/pipeline/syntax/:
There is quite a bit more, I just don't want to paste it all here. From their example, we have the following choices of parameter type: They are then used in the build process (Jenkinsfile from vyos-build again): Now we have multiple options, use a string or text field, and make the user add packages with comma/newline/space separation, then split and trim it and add it in a loop to the CUSTOM_PACKAGES. Done. Then you can flexibly add packages via Jenkins as well. Of course we could also add more stuff here, like additional package sources, etc. we'd of course then have to do a bit more modification of the build process, but it wouldn't be very hard to do. Like one text field for apt repo sources.list entries. Another one for priorities, if you want to build using your modified images from that other repo (that is how that works, right? I don't use apt-based systems that often). Then adding a build suffix in the filename, so you know what you've built. And all the while Jenkins does all the heavy lifting for you. |
Beta Was this translation helpful? Give feedback.
-
|
I use the debian live hooks (the I have script that watches for releases by vyos team (like they release 1.3.7), then it pull latest vyos-build for given branch, builds vyos-build docker image (applies some patches depending how vyos-build is broken, in past it was for example broken gem dependency for ruby), injects my shell script to The hooks are very easy, very flexible and very powerful - that's what the vyos-build uses to do everything - you can include some additional file, you can change some files, configure some things, you can install packages from whatever repository, you can build inside the chroot, basically you can anything and everything. This is why I don't even need the toml flavors, the live-build hooks are the most powerful tool to modify the imagine. Could you do this via Jenkins? You could but why would I? This is nice example to see how different people see different ways to be the best depending on where they did start. I started with my own ISO build thus I don't see the point why I would use Jenkins for that since I already have something that works as best it could, I'm using it for many years since the crux release, I know it has very low maintenance and infinite flexibility. Thus I see Jenkins as good fit for the packages part but not for the ISO or smoketest part - I can do that on my own just as well. Veduco started with everything so he doesn't have reason to use Jenskins at all. Perhaps you want to use Jenkins for everything because that's what did you started with. At the end you basically execute some script at some point and how you do it and how the script looks like doesn't really matter. Thus I see my half-Jenkins, yours full-Jenkins or Veduco's no-Jenkins as the same thing, just different people have different starting experiences and taste. I see how Jenkins saves you work but I also see how Jenkins imposes restrictions and some learning curve to overcome but for sure it's easier to learn than to learn what I did need to know to build my own solution and that's why I would recommend Jenkins for someone who starts from beginning and doesn't have experience with this sort of stuff but if you already have solution - should you migrate to Jenkins? No, you would just be changing things for the sake of changing them and that's pointless, you would exchange some advantage for another disadvantage. |
Beta Was this translation helpful? Give feedback.
-
|
... Like I said in my comment before last, I just create a different branch in If you now want to run I only wanted to point out, that it helps automate stuff. Were I to work on multiple branches in my All our approaches are entirely valid and I didn't express my ideas for customization as a way to discourage you to use yours. That is precisely why I added the line at the end
EDIT: @dd010101: In general, please always assume that me expressing my opinion is in no way a criticism of your way of working. I never mean to offend you and you seem to like taking things personally. |
Beta Was this translation helpful? Give feedback.
-
|
You explained your thoughts and so did I, so you don't take the things I say as against you. I would agree that what you saying about Jenkins is perfectly fine way to go about it. I wanted explain what I see as the best way to create customized ISO and to add some balance to the Jenkins discussion. |
Beta Was this translation helpful? Give feedback.
-
vyos-build repo
I would like to spare you some of the pain I went through:
equuleus.tar.txt
sagitta.tar.txt
current.tar.txt
Adjust the parts marked with TODO to match your system!
In theory you can apply them with
git amif I recall correctly, but as I've removed some stuff from the headers, you might need to usepatchinstead.vyos-strongswan repo
Clone the vyos-strongswan repo and apply this patch for the equuleus branch:
vyos-xe-guest-utilities repo
Furthermore the vyos-xe-guest-utilities repo has no sagitta branch, create it by copying the equuleus one.
Jenkins
Jenkins Global Configuration
/bin/bash/home/vyos_bld/.profile/home/vyos_bld/.profile/home/sentrium/web/dev.packages.vyos.net/public_html/repositoriestruehttps://yours-here:5000~/web/dev.packages.vyos.net/public_html/isosCreate the directories beforehand, otherwise it will probably fail.
Remember that docker containers don't inherit the
/etc/hostsof the host system, so when DNS names are used a DNS server must have the records.For the following special configuration is required:
I'm using the Git Filter by name (with wildcards) behaviour with Include set to
*and Exclude to:t*-* T*-* */* *feature* *patch* *PR* crux masterJenkins Plugins
Jenkins Plugins I installed:
Added also, though only 3-4 of these manually selected, the rest are dependencies, don't know if required or not:
Their Package versions
Some versions of packages that they used can't be built now as the dependencies of modules that are pulled in at build time, for example when running go builds, now conflict with older software versions present in the docker build environment.
Jenkinsfile.docker and Jenkinsfile for iso building
You can use the vyos-build's
Jenkinsfile.dockeras "Jenkinsfile" in a multi-branch project to build the docker vyos-build image. vyos-build'sJenkinsfilebuilds the iso. It can also be used. Make sure you select the Smoketests. I had quite a hard time to get it to not fail. They will run a long time, i mean plan in 2h+ for an iso + smoketest run.For me a workstation Xeon platform created nothing but problems with the threading/timing selftests that strongswan runs. Running it on an Epyc platform with all energy savings deactivated the problems were just gone.
Regarding the failures:
current smoketests
fail at TPM stage with
No module named 'vyos.util', which is due to the following missing patch to fix a bug:As I have no wish to also clone vyos-utils or vyos-1x or wherever that was a part of, I'll just wait till it is fixed.
sagitta smoketests
Ran fine the last run.
equuleus smoketests
sometimes fail due to:
DEBUG - test_dhcp_vrf (main.BondingInterfaceTest) ... FAIL
https://vyos.dev/T6025
T6025 smoketest: bonding: test fails every once in a while
Happens sometimes, reruns typically have no issues.
Out-of-memory
If the memory of the virtual machines started for smoketests is not big enough, you'll get out-of-memory errors.
Check that the ones I set aren't too high for your system. Also, it starts multiple VMs concurrently, so ... 10G of system memory for the build VM won't be enough. I just set it to 100G, gave it 32 cores and let it run ... No idea what the minimum is. There is also the bit about CPU cores the script detects and cuts in half. That was a bit rough in my case with 32 cores, so I set a hard limit (see patches).
Results
I was able to build both equueleus and sagitta stable images with smoketests succeeding. The equuleus one I installed on a R630 server and ran the smoketest successfully via:
Java + SSL Truststore
If you want to use SSL you might need to add your root CA to the Java truststore for all jenkins and build vms:
Localstack
Should you think about using localstack as a local AWS repository, stop now. Only the pro version allows for persistent data storage, otherwise everything you upload there is gone after a reboot/restart of localstack.
That is why I switched to ssh for iso upload.
Local docker registry
Both the basic authentication directly via the docker registry nor the nginx and apache2 recipes provided by the documentation have a functioning separation between docker push and pull requests, meaning one can't be unauthenticated while the other requires it. We'll have to disable it or add docker login commands to
vars/buildPackages.groove.I just disabled it.
Jenkins built agent VM
If you use a build agent VM, use SSH to start the agent and not the java .jar thing that Jenkins tells you to. That created some problems for me, though I can't recall what it was at the moment.
Iproute2
Regarding your question about the iproute2 stuff, they have a modified package that adds macsec commands (iirc). Otherwise functionality will be broken and smoketests related to it will fail.
Uncron
Used by the Jenkinsfile ssh commands to trigger reprepro jobs:
Well, I took two solutions you used that were more elegant than mine, I hope you won't mind. Worked them in there in the last two hours while removing hardcoded DNS entries and migrating to more Jenkins environment variables. So far it seems to be working, but that part I haven't tested extensively.
Beta Was this translation helpful? Give feedback.
All reactions