slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2855183	dotnet-runtime	Libraries Test Run checked coreclr osx x64 Release	Log
1222664	dotnet/runtime	osx-x64 Release Libraries_CheckedCoreCLR	Log
2855036	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1222459	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#121614
1222430	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#119432
1222411	dotnet/performance	Performance scenarios ubuntu 2204 x64 Open 8.0	Log
2854925	dotnet-performance	Performance roslyn ubuntu 2204 x64 Tiger 9.0	Log
1222350	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122249
1222239	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51068
1222266	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#122245
1222186	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122239
1222255	dotnet/performance	Performance blazor_scenarios ubuntu 2204 x64 Open 8.0	Log	dotnet/performance#5063
1222165	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT_RuntimeTests	Log
1222217	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2854793	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2854686	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1221563	dotnet/xharness	Helix Tests Build_Debug	Log
2854427	dotnet-maui	Build macOS	Log
1221425	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
1221481	dotnet/runtime	osx-x64 Debug PALTests	Log
1221457	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51826
1221435	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#121305
2854265	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2854229	dotnet-sdk	Darwin	Log	#55556
2854228	dotnet-sdk	Darwin	Log	#55558
2854227	dotnet-sdk	Darwin	Log	#55557
2854196	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2854191	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1221121	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#122206
2854049	dotnet-sdk	AoT: macOS (x64)	Log	#55844
2854139	dotnet-sdk	AoT: macOS (x64)	Log	#55853
1220996	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#122195
1221012	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
1221011	dotnet/performance	Performance roslyn ubuntu 2204 x64 Open 8.0	Log
2854067	dotnet-performance	Performance imagesharp ubuntu 2204 x64 Viper 9.0	Log
1220940	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#122094
1220920	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#122162
1220914	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122083
1220818	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1220908	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122200
1220893	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#48599
1220842	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#122085
1220734	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
1220758	dotnet/arcade	Windows_NT Build_Release	Log
1220744	dotnet/arcade	Windows_NT Build_Release	Log	dotnet/arcade#16371
2854008	dotnet-sdk	AoT: macOS (x64)	Log	#55853
1220569	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51068
1220713	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122197
1220741	dotnet/runtime	osx-arm64 Release Libraries_Release_CoreCLR	Log
1220737	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_RuntimeTests	Log
2853918	dotnet-sdk	TestBuild: macOS (x64)	Log	#55844
1220665	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono_Smoke	Log
1220604	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#120330
1220598	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1220575	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#122189
1220556	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#51995
1220544	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit		dotnet/runtime#121412
2853720	dotnet-maui	Build macOS	Log
1220383	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#121853
1220359	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122173
1220228	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122147
1220220	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#118932
1220069	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
1220152	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122174
1220056	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51881
1220098	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122173
1219939	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122170
1219739	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122154
1219660	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122164
1219470	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122045
1219528	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#121679
1219506	dotnet/runtime	osx-x64 Debug CoreCLR_Libraries	Log	dotnet/runtime#122157
1219426	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121927
1219384	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119634
1219362	dotnet/runtime	osx-arm64 Release Installer_Build_And_Test	Log
1219422	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
1218790	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#119352
1219292	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122072
1219301	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#117571
1219278	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122152
1219261	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122048
1219214	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122111
1219221	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122119
1219207	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121305
1219308	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1478
1219307	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1489
1219166	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122127
1219171	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122123
1219271	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#121853
1219137	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122147
1219142	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122149
1219104	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121412
1219225	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122119
1219071	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#122148
1219073	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#122148
1218841	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121739
1218856	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#121924
1218813	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#119575
1218607	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#122036
1217987	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#122094
Displaying 100 of 722 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
15	252	722

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.