Skip to content

Automated Video Upload Pipeline Documentation

Overview

This system is an event-driven architecture designed to automate the post-processing of video conferences. Specifically, when a recording session ends in LiveKit, the system automatically retrieves the video file from Cloudflare R2, uploads it to a specific YouTube Channel, and logs the activity to Google Sheets and Slack.

The system is hosted on Google Cloud Platform (GCP) using a hybrid approach of Cloud Run Services (for lightweight event listening) and Cloud Run Jobs (for heavy-duty processing). This separation of concerns ensures scalability, cost-efficiency, and reliability, preventing long-running uploads from timing out.

Infrastructure & Cloud Configuration

Setting up the infrastructure required navigating specific security policies and configuring a dedicated identity for the application.

The Google Cloud Project

The entire application resides within the video-uploader project on Google Cloud. We utilized the Artifact Registry to store our Docker images securely. A repository named containers was created in the us-central1 region to host the build artifacts generated by our CI/CD pipeline.

Service Account & Identity Management

To follow the principle of least privilege, we created a dedicated Service Account named video-sa. This identity is used by our CI/CD pipeline (GitHub Actions) to authenticate against Google Cloud without requiring human intervention.

We assigned specific IAM roles to video-sa to ensure it can perform exactly what is needed, and nothing more:

  • Artifact Registry Writer: Allows the account to push new Docker images to the registry
  • Cloud Run Admin: Allows the account to update and deploy the Service and the Job
  • Service Account User: Allows the account to "act as" itself when running the containers
  • Cloud Run Invoker: Grants permission to the Service to trigger the Job execution

Overcoming Organization Policies (The JSON Key Challenge)

During the setup, we encountered a strict Organization Policy enforced by Google Cloud by default: iam.disableServiceAccountKeyCreation. This policy prevents the creation of long-lived JSON keys to mitigate security risks.

To allow our GitHub Actions pipeline to authenticate, we had to explicitly override this policy at the project level:

  1. We navigated to the Organization Policies settings
  2. We located the constraints iam.disableServiceAccountKeyCreation (and its legacy counterpart iam.managed.disableServiceAccountKeyCreation)
  3. We set the enforcement to Off (Not Enforced)

This administrative action allowed us to generate a key.json file for the video-sa account, which serves as the "master key" for our deployment pipeline.

Authentication & The "YouTube Trio"

The system requires two distinct layers of authentication: one for the infrastructure (discussed above) and one for the application logic (talking to YouTube).

Because the Google Cloud Project (Infrastructure) and the YouTube Channel (Content) belong to two different Google Accounts, we implemented a specific OAuth2 flow.

The OAuth2 Application

On the Google Cloud project, we created an OAuth 2.0 Client ID (Desktop Application type). This generated two critical credentials:

  • Client ID: The public identifier of our application
  • Client Secret: The secret password proving ownership of the application

The Chain of Trust (Refresh Token)

To allow the application (Account B) to upload videos to the YouTube Channel (Account A), we performed a manual authorization exchange:

  1. We used the Google OAuth Playground
  2. We configured the playground to use our specific Client ID and Client Secret
  3. We logged in with Account A (the Channel Owner) and granted the youtube.upload scope permissions
  4. This generated a Refresh Token

This Refresh Token is the bridge. It allows our code to request temporary Access Tokens to upload videos forever, without requiring the channel owner to log in again.

System Architecture & Code Logic

The application logic is containerized using Docker and written in TypeScript (Node.js). It is deployed as a single Docker image that behaves differently based on the entry command (service.js vs job.js).

The Listener (service.ts)

This component is deployed as a Cloud Run Service. It acts as the "Receptionist."

  • Role: It exposes a public HTTP endpoint (/livekit/webhook) that listens for events

  • Logic:

    1. It receives a webhook from LiveKit
    2. It verifies the cryptographic signature using the LIVEKIT_API_KEY and LIVEKIT_API_SECRET to ensure the request is legitimate
    3. It filters for the egress_ended event (indicating a recording has finished)
    4. The Trigger: Instead of processing the video itself, it uses the Google Cloud API (googleapis) to invoke the Cloud Run Job
    5. Payload Injection: Crucially, it passes the webhook event data to the Job by injecting an environment variable override named JOB_PAYLOAD. It targets the specific container name (pipeline-1) to ensure the variable is read correctly

The Worker (job.ts)

This component is deployed as a Cloud Run Job. It acts as the "Heavy Lifter."

  • Role: It executes a long-running task and shuts down when finished. This avoids HTTP timeout limits (Cloud Run Services often timeout after 60 minutes, whereas Jobs can run for 24 hours)

  • Logic:

    1. Initialization: It starts up and looks for the JOB_PAYLOAD environment variable passed by the Listener
    2. R2 Retrieval: Using the AWS S3 SDK (compatible with Cloudflare R2), it connects to the bucket using R2 credentials and downloads the video file to the local container filesystem (/tmp). This ensures network stability during the upload process
    3. YouTube Upload: It authenticates using the "YouTube Trio" (Client ID, Secret, Refresh Token). It initiates a resumable upload session to YouTube, sets the privacy status to "Unlisted," and streams the file from the disk
    4. Notifications: Once the upload returns a Video ID, the worker uses the Slack API to post a message in the designated channel and the Google Sheets API to append a new row in the archive spreadsheet

Continuous Integration & Deployment (CI/CD)

We automated the entire build and deploy process using GitHub Actions. The workflow file (build-publish.yaml) ensures that changes to the dev branch are immediately reflected in production.

The Workflow Steps

  1. Trigger: The pipeline starts manually via workflow_dispatch (allowing control over when to deploy)
  2. Authentication: It authenticates with Google Cloud using the GCP_SA_KEY (the JSON key stored in GitHub Secrets)
  3. Build: It builds the Docker image from the root of the monorepo, ensuring all shared packages are included
  4. Push: It tags the image with the specific Git Commit SHA (for version tracking) and pushes it to the Google Artifact Registry
  5. Deploy Job: It updates the Cloud Run Job (video-upload-job) with the new image. It injects all necessary secrets (YouTube keys, R2 keys, Slack tokens) as environment variables
  6. Deploy Service: It updates the Cloud Run Service (livekit-listener) with the new image and sets it to allow unauthenticated public access (so LiveKit can reach it)
  7. Permissions: It re-applies the IAM policy binding to ensure the Service Account has permission to invoke the Job

Summary of Secrets & Keys

The system relies on the following secrets, securely stored in GitHub Actions:

  • Infrastructure:
    • GCP_SA_KEY: The JSON Service Account key for deployment
    • PROJECT_ID: The Google Cloud project ID
  • Application (Source):
    • LIVEKIT_API_KEY / SECRET: For webhook verification
    • R2_ACCESS_KEY / SECRET: For downloading raw video files
  • Application (Destination):
    • GOOGLE_CLIENT_ID / SECRET: The App credentials
    • GOOGLE_REFRESH_TOKEN: The authorization for the YouTube Channel
    • SLACK_BOT_TOKEN: For sending chat notifications
    • GOOGLE_SHEET_ID: For logging the database record

Conclusion

This architecture provides a robust, serverless solution for video processing. By decoupling the "listening" phase from the "processing" phase, we ensure that the system can handle large video files without timing out, while the Organization Policy overrides and rigorous IAM configuration ensure the system is both secure and automated.