Cleaning Up Orphaned CI Processes and Port Conflicts on Cloud Macs

After multiple CI runs on the same cloud Mac, the hardest failures to explain are often not compilation errors, but cases where “the previous run has finished, yet the next one cannot start.” A test service reports that its port is already in use, simulator instances keep accumulating, build directories cannot be removed, or multiple background processes with the same name write to the same log. Restarting the machine may provide temporary relief, but it masks the underlying question of resource ownership. A more reliable approach is to collect evidence first, then keep every run’s resources within boundaries that can be identified and reclaimed.

Identify Who Owns the Leftover Resources

When you see Address already in use, do not immediately run a global killall. Identify the listening process first, then follow its parent process chain to determine whether it belongs to a completed job.

PORT=8080
lsof -nP -iTCP:"$PORT" -sTCP:LISTEN
ps -o pid=,ppid=,user=,lstart=,command= -p 41872
ps -o pid=,ppid=,user=,command= -p 41801

After lsof returns a PID, verify at least the user, start time, full command, and PPID. If the parent process is still the current Runner, the port may belong to a parallel job that is still running. If the parent has become a system process, that alone does not mean the process is safe to remove, because a background task may have deliberately detached from its original session.

Generate a RUN_ID for every run and include it in process arguments, log paths, or environment variables. During investigation, use the following commands to narrow the search:

RUN_ID="${CI_RUN_ID:-local-$(date +%s)}"
export XCODEVM_RUN_ID="$RUN_ID"
pgrep -afil "$RUN_ID"

Cleanup should be based on a resource identifier matching a completed job, not on a process name that merely looks familiar. Parallel jobs on the same machine may execute exactly the same command.

Give Every Run Its Own Boundaries

Shared directories and fixed ports turn occasional interruptions into persistent failures. Build directories, result bundles, temporary files, and service ports should all be isolated by run.

xcodebuild lets you specify the build data and result paths explicitly:

WORK_ROOT="${CI_WORK_ROOT:-$HOME/ci-work}"
RUN_ROOT="$WORK_ROOT/$RUN_ID"
DERIVED_DATA="$RUN_ROOT/DerivedData"
RESULT_PATH="$RUN_ROOT/Test.xcresult"

mkdir -p "$RUN_ROOT"
xcodebuild test \
  -scheme App \
  -derivedDataPath "$DERIVED_DATA" \
  -resultBundlePath "$RESULT_PATH"

Do not let two concurrent jobs share the same Derived Data directory. Even when the project and branch are identical, intermediate artifacts, index databases, and cleanup operations can still race with one another.

Use Exit Traps for Failures and Interruptions

Running cleanup only at the end of a script is not enough. Compilation failures, timeout signals, and manual cancellations can all bypass the final lines. A shell trap can cover normal exits and common interruption paths consistently.

set -euo pipefail

RUN_ID="${CI_RUN_ID:-local-$(date +%s)}"
RUN_ROOT="$(mktemp -d "${TMPDIR:-/tmp}/xcodevm-${RUN_ID}.XXXXXX")"
SIM_UDID=""
SERVICE_PID=""

cleanup() {
  if [[ -n "$SERVICE_PID" ]] && kill -0 "$SERVICE_PID" 2>/dev/null; then
    kill -TERM "$SERVICE_PID" 2>/dev/null || true
    for _ in 1 2 3 4 5; do
      kill -0 "$SERVICE_PID" 2>/dev/null || break
      sleep 1
    done
    kill -KILL "$SERVICE_PID" 2>/dev/null || true
  fi

  if [[ -n "$SIM_UDID" ]]; then
    xcrun simctl shutdown "$SIM_UDID" 2>/dev/null || true
    xcrun simctl delete "$SIM_UDID" 2>/dev/null || true
  fi

  rm -rf "$RUN_ROOT"
}

trap cleanup EXIT INT TERM

This records the PID and UDID created by the current run instead of scanning for and deleting every resource with the same name. TERM gives the process an opportunity to release file handles and finish writing logs. Escalate to KILL only if it still has not exited after the wait period.

Avoid Killing a Reused PID

During long-running jobs, the system may reassign an old PID. Before terminating a process, read its command line again and verify that it contains the run identifier. A safer design is for the background service to write its PID, start time, and a command summary to $RUN_ROOT/manifest, then require all of those fields to match during cleanup.

Handle Simulators and Service Managers

Simulators should not be managed by name alone because devices with the same name may belong to other jobs. Save the UDID immediately after creation:

SIM_UDID="$(xcrun simctl create "ci-$RUN_ID" \
  "iPhone 16" \
  "com.apple.CoreSimulator.SimRuntime.iOS-18-0")"
xcrun simctl boot "$SIM_UDID"

If the required Runtime is not installed in the execution environment, use xcrun simctl list runtimes to obtain an identifier that is actually available. Do not hard-code the version in shared scripts.

If a process automatically reappears after exiting, check whether it is managed as a user-level service. Inspect the current user domain first, and do not immediately unload a service whose origin is unknown:

USER_ID="$(id -u)"
launchctl print "gui/$USER_ID" | grep -B 3 -A 6 "$RUN_ID"

If the job did create a temporary service, save its label and use that same label to remove it precisely during the exit path. Deleting only the process PID is often ineffective because the service manager will launch it again.

Build a Repeatable Verification Checklist

Completing cleanup does not necessarily mean the problem is resolved. Run the same job twice in succession and verify after each run that resources have returned to baseline:

1. Use lsof to confirm that the test port is no longer listening.
2. pgrep -afil "$RUN_ID" should not return any background process from the current run.
3. The name and UDID created by the current run should no longer appear in xcrun simctl list devices.
4. The temporary run directory has been deleted, and any logs or result bundles that must be archived were first copied to persistent storage.
5. The second run uses a new run identifier and does not depend on caches or services left by the first run.
6. When concurrent jobs are present, clean up only the resources recorded in the current job’s manifest.

If the port remains occupied, preserve the lsof output, two levels of parent processes, the start time, and the command line before deciding whether the cause is incomplete cleanup or legitimate concurrency. If the issue occurs only after forced cancellation, focus on whether signals are propagated to the wrapper script and whether background processes create independent sessions. Incorporating these checks into the Runner’s finalization stage is safer and more reproducible than periodically scanning for processes and terminating them in bulk.

Frequently asked questions