Hi and welcome to my blog, where I’ll share the experience I had troubleshooting connecting to private NuGet feed across Azure DevOps organizations
Recently, I took on a project that has allowed me to work with .NET and delve deeper into the world of DevOps. Follow along as I share the challenges and lessons learned while working on this project.
Why .NET?
As someone who is always looking to learn and improve, I was excited to take on a new project that required me to update an existing backend built with Node.js and Typescript Express. We chose to use .NET for this project because of its strong support for object-oriented programming, active developer community, and seamless integration with Microsoft Azure. Azure offers a range of services and tools for building, deploying, and managing applications, making it an excellent choice for our needs. Overall, we believed that .NET is the perfect fit for this project and were excited to see the results of our efforts.
Using templates from different Azure DevOps organization
One of my first tasks on the project was setting up a CI/CD pipeline for .NET projects using Azure DevOps. The pipeline had to use pipeline templates created by another team who already used .NET, meaning we had to use templates from a git repository hosted on a different Azure organization (because each team is a separate Azure DevOps organization). These templates were great and easily extendable to our use case without breaking anything for the team that owned them.
After figuring out how to use the templates, I started writing our base pipeline and setting up the necessary service connectors. Service connectors are a way to securely connect resources across organizations and services so that they can be accessed inside pipelines. However, while setting up the pipeline, I ran into issues with accessing shared NuGet packages that were stored in a feed on the other team’s Azure DevOps. I followed the documentation and created a service connector between the two organizations, and added a reference to the connector in the pipeline’s YAML. Despite these efforts, I was still unable to access the artefacts and the pipeline kept failing.
This was a real pain in the ass because I was running and running and running the pipeline, and each time it was taking a few minutes to run, which is always a problem when debugging a CI/CD pipeline. I was becoming increasingly frustrated as the pipeline kept failing and I was wasting a lot of time. Despite knowing that I needed to find a solution, I wasn’t sure where to begin and I felt stuck and unable to get the pipeline working.
Microsoft’s NuGet Authenticate task for CI pipelines
The task I’ve had an issue with is Microsoft’s NuGet Authenticate task for CI pipelines. Using the task is usually as simple as adding this step into your CI pipeline yaml:
steps:
- task: NuGetAuthenticate@0
displayName: 'Authenticate with NuGet'
In my case that wasn’t enough as the packages came from another organization. To access the necessary packages from another organization, I set up a temporary service connector within Azure DevOps using my personal access token. I thought to myself: “Great, now I can just reference it in my yaml template and that’s it!” I’ve added the option as a parameter so that it doesn’t break compatibility with other team’s pipelines, as they are owners of these templates.
parameters:
- name: nuGetServiceConnections
displayName: Optional additional organizations nuget service connections
type: string
default: 'default'
steps:
- task: NuGetAuthenticate@0
displayName: 'Authenticate with NuGet'
${{ if ne(parameters.nuGetServiceConnections, 'default') }}:
inputs:
nuGetServiceConnections: ${{ parameters.nuGetServiceConnections }}So I ran the pipeline, and it failed. Made a few changes, trying to eliminate as many variables as possible, and it failed again. Then again. Then again and again and again… Quickly my git branch looked like this:
This went on for days and days. It was particularly frustrating because it only occurred when using a container with the dotnet restore command, but not when using the build outside of a container for testing. What was even more frustrating, was not knowing if the issue is even with the pipeline or with the docker file itself, so I’ve been troubleshooting that in different ways, trying to reduce the number of possible reasons and places where the failure occurs.
Going back to the documentation
During the build process, the NuGetAuthenticate task creates a secret token. When I attempted to print this token, it was replaced with ‘***’ in the output, indicating that Azure was successfully hiding sensitive information. This was a good sign, as it meant that there was a valid token inside. However, I encountered an issue when using Docker to build the project. I was able to build it locally using my Azure DevOps personal access token (PAT), but I couldn’t get it to work in the pipeline. I examined both the NuGetAuthenticate@1 documentation and the Docker build command documentation a thousand times now, but I couldn’t find the cause of the issue. The VSS_NUGET_ACCESSTOKEN variable which should be set is set, it is correctly sent to docker file, yet the build fails inside docker but succeeds outside.
Finding the token inside VSS_NUGET_EXTERNAL_FEED_ENDPOINTS
After several days of trying to troubleshoot the issue, I decided to check the source code for the NugetAuthenticate task on GitHub as my last desperate attempt. To my relief, I discovered that Microsoft makes the source code for their Azure DevOps tasks publicly available. I was able to locate the NugetAuthenticate task and examine its code to see what it was adding to the environment variables.
While examining the source code for the NugetAuthenticate task, I noticed that it was only setting the VSS_NUGET_ACCESSTOKEN environment variable for authenticating with the NuGet feed of the organization that the task belonged to. This seemed strange to me, as there was also a parameter available for authenticating with different organizations. However, I discovered that the task also set the VSS_NUGET_EXTERNAL_FEED_ENDPOINTS environment variable, which contained a JSON object with a token for each connector defined in the task’s parameters. This helped me to successfully build the Docker container for my project, as I was able to extract the necessary token from the JSON object using a bit of regular expression magic. Despite the initial difficulties, I was ultimately able to finish my task and move on to the next step.
Final nuget_auth.yaml used as part of my pipeline ended up looking like this:
parameters:
- name: nuGetServiceConnections
displayName: Optional additional organizations nuget service connections
type: string
default: 'default'
steps:
- task: NuGetAuthenticate@0
displayName: 'Authenticate with NuGet'
${{ if ne(parameters.nuGetServiceConnections, 'default') }}:
inputs:
nuGetServiceConnections: ${{ parameters.nuGetServiceConnections }}
- bash: |
# extract token from service connector nuget auth endpoints json
token=$(echo $(VSS_NUGET_EXTERNAL_FEED_ENDPOINTS) | sed -e 's/.*password:\(.*\)\]\}/\1/')
# create pipeline variable
echo "##vso[task.setvariable variable=VSS_NUGET_EXTERNAL_FEED_ENDPOINTS_TOKEN]$token"
displayName: 'Extract service connector token for nuget feed from env variable'
Lesson learned
Read the documentation, but don’t trust it blindly. People lie, code doesn’t. 🙂
