Debugging Node, Pnpm & Dependencies: Setup & Workflow Tips

Hey guys! We've got a bit of a situation with our CI builds being flaky. It seems like pnpm isn't always available when our steps are running, and the order of actions in our .github/workflows/ci.yml file isn't helping. Plus, we don't have a straightforward way to check our local and CI environments for the correct versions of Node, npm, pnpm, and Nx. So, let's dive into how we can fix this and make our lives easier!

Understanding the Problem

Currently, we're seeing errors like "Unable to locate executable file: pnpm" in our CI. This tells us that pnpm, which is our package manager, isn't being installed or recognized before it's needed. Our caching mechanism, set up in actions/setup-node, isn't working as expected because pnpm isn't installed yet. And to top it off, our contributors might be using different versions of tools, leading to inconsistencies and more headaches.

Goal

Our goal here is to streamline our CI process and ensure everyone is working with the same tools. We'll tackle this by:

1. Fixing the GitHub Actions workflow to install pnpm before caching Node.
2. Making sure our jobs run in parallel (lint, typecheck, test, build).
3. Pinning versions sanely and making caching deterministic.
4. Adding a handy verify-env.sh script to validate our environment.

Let's break down the steps to achieve this.

1. Correcting the GitHub Actions Workflow

Setting Up pnpm Before Node Caching

The core of the issue lies in our .github/workflows/ci.yml file. We need to ensure pnpm is set up before we cache Node. To do this, we'll use the pnpm/action-setup@v4 action before actions/setup-node@v4. This ensures pnpm is available and can be cached correctly.

- uses: pnpm/action-setup@v4
  with:
    version: 8 # Or your desired pnpm version
- uses: actions/setup-node@v4
  with:
    node-version: '20'
    cache: 'pnpm'

This snippet first sets up pnpm with a specific version (you can adjust this as needed) and then sets up Node, using pnpm for caching. This is crucial for consistent builds.

Running Jobs in Parallel

To speed up our CI process, we'll run our linting, type checking, testing, and building jobs in parallel. However, the build job should only run after the others have completed successfully. We can define these dependencies in our workflow file.

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - name: Lint
        run: | 
          echo "Lint Job"
  typecheck:
    runs-on: ubuntu-latest
    steps:
      - name: Typecheck
        run: |
          echo "Typecheck Job"
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Test
        run: |
          echo "Test Job"
  build:
    needs: [lint, typecheck, test]
    runs-on: ubuntu-latest
    steps:
      - name: Build
        run: |
          echo "Build Job"

In this setup, lint, typecheck, and test jobs will run concurrently, and the build job will wait for all of them to finish. This parallelism can significantly reduce CI time.

Adding Timeouts, Concurrency, and Permissions

To prevent jobs from running indefinitely and to manage concurrent runs, we'll add timeouts, concurrency settings, and permissions to our workflow.

jobs:
  lint:
    timeout-minutes: 10
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Lint
        run: |
          echo "Lint Job"
concurrency:
  group: ci
  cancel-in-progress: true

Here, we've set a timeout of 10 minutes for the lint job, given it read permissions for the repository contents, and configured concurrency to cancel any in-progress runs when a new push is made. This helps keep our CI clean and efficient.

Setting Fetch Depth and Copying Environment Variables

To future-proof our workflow and ensure Nx can correctly determine affected projects, we'll set fetch-depth: 0 on the checkout action. Additionally, we'll copy our .env.example file to .env in jobs that need it.

- uses: actions/checkout@v4
  with:
    fetch-depth: 0
- name: Copy .env
  run: cp .env.example .env

This ensures we fetch the full history and have access to necessary environment variables.

Uploading Artifacts

Finally, we'll upload artifacts from our web, docs, and Storybook builds. This allows us to easily access build outputs for deployments or debugging.

- name: Upload Artifacts
  if: always()
  uses: actions/upload-artifact@v3
  with:
    name: web-build
    path: apps/web/.next
    retention-days: 7

We're uploading the .next directory from our web app, the build directory from our docs app, and the storybook-static directory from our Storybook app, retaining them for seven days. These artifacts can be invaluable for troubleshooting and deployments.

2. Version Pinning and Environment Consistency

Pinning pnpm Version in package.json

To ensure consistent environments, we'll pin the pnpm version in our package.json file. If it's missing, we'll add `