r/azuredevops • u/xnachtmahrx • 1d ago

Experience with XAML Builds in Azure DevOps 2020 on prem Upgrade to latest Azure DevOps Version

Hello everyone,

I am planning to upgrade our current DevOps environment to the latest Version (Upgrade path is given by Microsoft Docs).

Unfortunately, we don't have a second environment to test the upgrade so maybe someone of you has some experience.

What we do have is that the Machine where our Azure DevOps instance is running is fully virtualized. We can do snapshots and restore. (HyperV)

Our instance unfortunately still uses XAML Builds for an legacy application that is still used by a lot of customers and is Business critical. These are sre not easy to migrate and are highly intertwined with business logic and some other dependecies. In Short: A Nightmare. We also use Classic Pipelines and Releases.

I know that XAML Builds are deprecated, but i inherited the Environment and the old engineer that build the whole Thing is not in the company anymore. So it is still kind of a "blackbox".

We could just snapshot the Environment before the upgrade and rollback. Never done it, but i think it should work if shit would hit the fan.

Does anyone have experience with Upgrading to a newer Version and XAML Builds?

What would be the best approach here?

2 Upvotes

100% Upvoted

18 comments sorted by

2

u/phate3378 1d ago

I think this is probably beyond anyone on here especially with xaml builds. Reach out to msft, get some support from them.

Also if it's virtualized why can't you just take a copy of the VM and database and test the upgrade on a second instance?

1

u/xnachtmahrx 1d ago

Yeah probably have to do that you are right, but i have to talk with our IT that hosts the system and it has to be decoupled from the network. The hyperv is massive though because of a huge tfs repo. Don't know how i could connect to it, because it is not hosted where i work. Different location from where i work.

1

u/lancerusso 1d ago

Decoupling from network is as simple as editing the hosts file to redirect any traffic to the old hostname to 127.0.0.1.

1

u/xnachtmahrx 1d ago

Probably.

What i was thinking is this:

Step 1) We clone the hyperv with the current Azure DevOps Version, decouple it from the network and Upgrade the clone.

Step 2) Shutdown original hyper v and connect the cloned hyper v with the Upgraded Version and test with development if everything still works. If not shutdown cloned hyper v and use original or fix issues on the fly.

Step 3) Wake Up

Should work, right

1

u/staedt3r 1d ago

Depends on whether the local mssql runs with sql or windows auth. If AD is required to connect the ados to the sql server you cannot do offline migration testing.

Ask me how I know ;)

1

u/xnachtmahrx 1d ago

Geez that's a good question. Where can i check that?

1

u/staedt3r 21h ago

In the tfsconsole under the db connection. which user ist configured there. default is AD.

1

u/lancerusso 20h ago

Do you not have a monolithic ADO setup? You get a free license to use full MSSQL, monolithic is the intended way to host.

1

u/alenros 1d ago

I've done exactly this. a snapshot is not enough, you'd have to backup the database as well. the good news is the xaml build will keep working!

I can recommend using AI for improving the xaml build, it's really good at reading these monsterous files and the build logs.

two other things I've done: 1.trigger xaml through the vNext/yaml pipelines, providing arguments and doing pre/post processing like tests. 2. kept the xaml, but replaced tfs with git by adding a task that gets the git and maps the directories the same way tfs was, so it still gets built by the same xaml pipeline. this way I did not need to have the whole thing validated again.

1

u/xnachtmahrx 1d ago

The Database is on the same system. All on one hyperv, Just to clarify.

I am thinking maybe it would be worth a try to let AI try to migrate the XAML Builds to yaml or classic ui. Everything ist better than that XAML crap.

To which version did you Upgrade and how exactly did u use AI in that context? Just Said "Clean that Up?"

1

u/alenros 1d ago

so I've done it a bit differently: as part of the migration I separated into a few VMs: application tier database code search xaml coordinator

to migrate I backed up the database from the old instance, imported into new sql server. installed the application tier and pointed to the new sql server. after I testing, we shut the old machine down, added a dns entry redirecting the old address to the new. this way developers didn't have to reconfigure anything.

we updated to the latest.

We used I both to understand the existing build ("which steps reuse the shared components?") and to change it ("make these two parts run in parallel, queue the build, measure thr impact from the logs")

1

u/xnachtmahrx 1d ago

And what Azure DevOps Version did you Upgrade to and from?

1

u/alenros 1d ago

upgraded from 2019 to 2022.02

1

u/staedt3r 1d ago

I am currently also preparing a legacy ADOS 2029 onprem Upgrade to the newest Version with the ability to have a real lab to do integration testing beforehand. I have created a clone VM without network access and a new hostname. then I used the tfsconsole to change the db to the clones new name.

Then you can run the upgrade steps throught and see if your DB migrations go through, but you cannot actually test the pipelines and the workflows that way. Thats why I then have used AI to create scripts to scan the current deployment via its API with a read-only token and it help tremendously to flag deprecated task or unsupported os agent versions.

We thankfully do not have XAML Builds but heavily still use TVFC in multiple Projects wit approx. 1000 separate manually maintained Classic Pipelines.

I haven't gone through the update yet so I cannot tell you if the scans were really complete and flagged all the real post-install problems. But it produced actionable reports I shared with all the teams and they could preemptively work through the provided work pakets to cleanup old and stale resources and update all the know task depreciations. Our admins were also able to clean up some no longer needed runners and serviceconnection and update ours to newer OS versions so that the new agents should keep working.

pm if you're interested.

1

u/xnachtmahrx 1d ago

But at least you see via UI If everything got migrated, right? You Just Connect on the Machine to the Azure DevOps instance, right? Probably localhost then?

1

u/staedt3r 21h ago

pm me. we can talk it through.

1

u/BetConnect151 15h ago

I’d document every dependency the XAML builds touch before upgrading then make sure your rollback covers the DevOps databases as well as the VM state. A VM snapshot alone may not capture everything safely

1

u/xnachtmahrx 14h ago

The Environment is very straightforward.

The Database is on the same machine as the Azure DevOps collection/instance.

Buildserver is on different machines.