Skip to content

feat(github-app): use ssm manifest for extra apps #5282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
guicaulada wants to merge 2 commits into
base: main
Choose a base branch
from
+351 −353
Open

feat(github-app): use ssm manifest for extra apps #5282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
071e4be
feat(github-app): use ssm manifest for extra apps
guicaulada Aug 19, 2026
70c81ad
docs: auto update terraform docs
github-actions[bot] Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/rate-limits-and-tuning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,17 @@ Sustained: 10,000/hour ÷ 60 = **~166 runners/minute**.

Without a token cache, each runner also costs a `POST /app/installations/{id}/access_tokens` (5 points) against the `core` endpoint. This doesn't directly reduce JIT throughput (different endpoint) but competes with `isJobQueued` for the `core` hourly budget.

### Distributing load across multiple GitHub Apps

Rate limits are per App installation and cannot be raised. To scale beyond one App's budget, configure extra Apps with `additional_github_apps`. The control-plane lambdas select one App per invocation, making the effective limit N × the per-App limit.

> [!IMPORTANT]
> Every additional App must be installed on the same organizations or repositories as the primary App. The module cannot verify this. A missing installation surfaces at runtime as installation lookup 404s on the fraction of invocations that select the misconfigured App, which is hard to trace back to the installation.

Only the primary App needs a webhook configured in GitHub; additional Apps are used for API calls only. Set `installation_id` per additional App to skip one installation lookup per invocation.

The lambdas receive additional App credentials through a manifest SSM parameter that lists the per-App credential parameter names, so the lambda environment size stays constant regardless of App count.

### GHES

Rate limits are **disabled by default** on GitHub Enterprise Server and must be explicitly enabled by the site admin. When enabled, the same formula applies.
Expand Down
48 changes: 29 additions & 19 deletions lambdas/functions/control-plane/src/github/auth.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { createAppAuth } from '@octokit/auth-app';
import { StrategyOptions } from '@octokit/auth-app/dist-types/types';
import { request } from '@octokit/request';
import { RequestInterface, RequestParameters } from '@octokit/types';
import { getParameters } from '@aws-github-runner/aws-ssm-util';
import { getParameter, getParameters } from '@aws-github-runner/aws-ssm-util';
import { generateKeyPairSync } from 'node:crypto';
import * as nock from 'nock';

Expand Down Expand Up @@ -35,6 +35,7 @@ const PARAMETER_GITHUB_APP_ID_NAME = `/actions-runner/${ENVIRONMENT}/github_app_
const PARAMETER_GITHUB_APP_KEY_BASE64_NAME = `/actions-runner/${ENVIRONMENT}/github_app_key_base64`;

const mockedGetParameters = vi.mocked(getParameters);
const mockedGetParameter = vi.mocked(getParameter);

beforeEach(() => {
vi.resetModules();
Expand Down Expand Up @@ -341,23 +342,32 @@ describe('Test getStoredInstallationId', () => {
vi.mocked(createAppAuth).mockReturnValue(mockWithHook);
});

it('returns stored installation ID when configured', async () => {
const installationIdParam = `/actions-runner/${ENVIRONMENT}/github_app_installation_id`;
process.env.PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = installationIdParam;
it('returns stored installation ID when configured for an additional app', async () => {
const appIdParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_id`;
const appKeyParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_key_base64`;
const installationIdParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_installation_id`;
process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME = `/actions-runner/${ENVIRONMENT}/additional_github_apps_manifest`;
mockedGetParameter.mockResolvedValueOnce(
JSON.stringify([
{ idParamName: appIdParam, keyParamName: appKeyParam, installationIdParamName: installationIdParam },
]),
);
mockedGetParameters.mockResolvedValueOnce(
new Map([
[PARAMETER_GITHUB_APP_ID_NAME, GITHUB_APP_ID],
[PARAMETER_GITHUB_APP_KEY_BASE64_NAME, b64],
[appIdParam, '2'],
[appKeyParam, b64],
[installationIdParam, '12345'],
]),
);

const result = await getStoredInstallationId(0);
const result = await getStoredInstallationId(1);
expect(result).toBe(12345);
});

it('returns undefined when installation ID param is empty', async () => {
process.env.PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = '';
it('returns undefined when the manifest env var is empty', async () => {
process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME = '';
mockedGetParameters.mockResolvedValueOnce(
new Map([
[PARAMETER_GITHUB_APP_ID_NAME, GITHUB_APP_ID],
Expand All @@ -369,8 +379,8 @@ describe('Test getStoredInstallationId', () => {
expect(result).toBeUndefined();
});

it('returns undefined when env var is not set', async () => {
delete process.env.PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME;
it('returns undefined when the manifest env var is not set', async () => {
delete process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME;
mockedGetParameters.mockResolvedValueOnce(
new Map([
[PARAMETER_GITHUB_APP_ID_NAME, GITHUB_APP_ID],
Expand All @@ -383,7 +393,7 @@ describe('Test getStoredInstallationId', () => {
});

it('returns undefined for out-of-bounds appIndex', async () => {
process.env.PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = '';
delete process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME;
mockedGetParameters.mockResolvedValueOnce(
new Map([
[PARAMETER_GITHUB_APP_ID_NAME, GITHUB_APP_ID],
Expand All @@ -395,21 +405,21 @@ describe('Test getStoredInstallationId', () => {
expect(result).toBeUndefined();
});

it('loads installation IDs for multi-app setup', async () => {
const app1IdParam = `/actions-runner/${ENVIRONMENT}/github_app_id`;
it('loads installation IDs for multi-app setup from the manifest', async () => {
const app2IdParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_id`;
const app1KeyParam = `/actions-runner/${ENVIRONMENT}/github_app_key_base64`;
const app2KeyParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_key_base64`;
const app2InstallParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_installation_id`;

process.env.PARAMETER_GITHUB_APP_ID_NAME = `${app1IdParam}:${app2IdParam}`;
process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME = `${app1KeyParam}:${app2KeyParam}`;
process.env.PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME = `:${app2InstallParam}`;

process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME = `/actions-runner/${ENVIRONMENT}/additional_github_apps_manifest`;
mockedGetParameter.mockResolvedValueOnce(
JSON.stringify([
{ idParamName: app2IdParam, keyParamName: app2KeyParam, installationIdParamName: app2InstallParam },
]),
);
mockedGetParameters.mockResolvedValueOnce(
new Map([
[app1IdParam, '1'],
[app1KeyParam, b64],
[PARAMETER_GITHUB_APP_ID_NAME, '1'],
[PARAMETER_GITHUB_APP_KEY_BASE64_NAME, b64],
[app2IdParam, '2'],
[app2KeyParam, b64],
[app2InstallParam, '67890'],
Expand Down
62 changes: 45 additions & 17 deletions lambdas/functions/control-plane/src/github/auth.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import { Octokit } from '@octokit/rest';
import { retry } from '@octokit/plugin-retry';
import { throttling } from '@octokit/plugin-throttling';
import { createChildLogger } from '@aws-github-runner/aws-powertools-util';
import { getParameters } from '@aws-github-runner/aws-ssm-util';
import { getParameter, getParameters } from '@aws-github-runner/aws-ssm-util';
import { EndpointDefaults } from '@octokit/types';

const logger = createChildLogger('gh-auth');
Expand Down Expand Up @@ -77,48 +77,76 @@ interface GitHubAppCredential {

let appCredentialsPromise: Promise<GitHubAppCredential[]> | null = null;

// One entry per additional app in the manifest parameter. The manifest keeps
// the lambda environment size constant regardless of the number of apps: the
// environment carries only the manifest's parameter name, and the manifest
// value lists the per-app credential parameter names.
interface AdditionalAppManifestEntry {
idParamName: string;
keyParamName: string;
installationIdParamName?: string | null;
}

async function loadAppCredentials(): Promise<GitHubAppCredential[]> {
if (!process.env.PARAMETER_GITHUB_APP_ID_NAME) {
const idParamName = process.env.PARAMETER_GITHUB_APP_ID_NAME;
const keyParamName = process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME;
if (!idParamName) {
throw new Error('Environment variable PARAMETER_GITHUB_APP_ID_NAME is not set');
}
if (!process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME) {
if (!keyParamName) {
throw new Error('Environment variable PARAMETER_GITHUB_APP_KEY_BASE64_NAME is not set');
}
const idParams = process.env.PARAMETER_GITHUB_APP_ID_NAME.split(':').filter(Boolean);
const keyParams = process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME.split(':').filter(Boolean);
const installationIdParams = (process.env.PARAMETER_GITHUB_APP_INSTALLATION_ID_NAME || '').split(':');
if (idParams.length !== keyParams.length) {
throw new Error(`GitHub App parameter count mismatch: ${idParams.length} IDs vs ${keyParams.length} keys`);

const entries: { id: string; key: string; installationId?: string }[] = [{ id: idParamName, key: keyParamName }];

const manifestParamName = process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME;
if (manifestParamName) {
const manifest = JSON.parse(await getParameter(manifestParamName)) as AdditionalAppManifestEntry[];
entries.push(
...manifest.map((entry) => ({
id: entry.idParamName,
key: entry.keyParamName,
installationId: entry.installationIdParamName ?? undefined,
})),
);
}

// Batch fetch all SSM parameters in a single call to reduce API calls
const allParamNames = [...idParams, ...keyParams, ...installationIdParams.filter((p) => p.length > 0)];
const allParamNames = entries.flatMap((entry) => [
entry.id,
entry.key,
...(entry.installationId ? [entry.installationId] : []),
]);
const params = await getParameters(allParamNames);

const credentials: GitHubAppCredential[] = [];
for (let i = 0; i < idParams.length; i++) {
const appIdValue = params.get(idParams[i]);
for (const entry of entries) {
const appIdValue = params.get(entry.id);
if (!appIdValue) {
throw new Error(`Parameter ${idParams[i]} not found`);
throw new Error(`Parameter ${entry.id} not found`);
}
const appId = parseInt(appIdValue, 10);
const privateKeyBase64 = params.get(keyParams[i]);
const privateKeyBase64 = params.get(entry.key);
if (!privateKeyBase64) {
throw new Error(`Parameter ${keyParams[i]} not found`);
throw new Error(`Parameter ${entry.key} not found`);
}
// replace literal \n characters with new lines to allow the key to be stored as a
// single line variable. This logic should match how the GitHub Terraform provider
// processes private keys to retain compatibility between the projects
const privateKey = Buffer.from(privateKeyBase64, 'base64').toString().replace(/\\n/g, '\n');
const installationIdParam = installationIdParams[i];
const installationIdValue =
installationIdParam && installationIdParam.length > 0 ? params.get(installationIdParam) : undefined;
const installationIdValue = entry.installationId ? params.get(entry.installationId) : undefined;
const installationId = installationIdValue ? parseInt(installationIdValue, 10) : undefined;
credentials.push({ appId, privateKey, installationId });
}
logger.info(`Loaded ${credentials.length} GitHub App credential(s)`);
return credentials;
}

export async function getLoadedAppId(appIndex: number): Promise<number | undefined> {
const credentials = await getAppCredentials();
return credentials[appIndex]?.appId;
}

function getAppCredentials(): Promise<GitHubAppCredential[]> {
if (!appCredentialsPromise) appCredentialsPromise = loadAppCredentials();
return appCredentialsPromise;
Expand Down
120 changes: 25 additions & 95 deletions lambdas/functions/control-plane/src/github/rate-limit.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,13 @@ import { ResponseHeaders } from '@octokit/types';
import { createSingleMetric } from '@aws-github-runner/aws-powertools-util';
import { MetricUnit } from '@aws-lambda-powertools/metrics';
import { metricGitHubAppRateLimit } from './rate-limit';
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
import { getParameter } from '@aws-github-runner/aws-ssm-util';
import { describe, it, expect, beforeEach, vi } from 'vitest';
import { getLoadedAppId } from './auth';

process.env.PARAMETER_GITHUB_APP_ID_NAME = 'test';
vi.mock('@aws-github-runner/aws-ssm-util', async () => {
// Return only what we need without spreading actual
return {
getParameter: vi.fn((name: string) => {
if (name === process.env.PARAMETER_GITHUB_APP_ID_NAME) {
return '1234';
} else {
return '';
}
}),
};
});
vi.mock('./auth', async () => ({
// App ids per index, as loaded by the auth module from SSM.
getLoadedAppId: vi.fn(async (appIndex: number) => [1234, 5678][appIndex]),
}));

vi.mock('@aws-github-runner/aws-powertools-util', async () => {
// Provide only what's needed without spreading actual
Expand All @@ -44,7 +35,6 @@ describe('metricGitHubAppRateLimit', () => {
});

it('should update rate limit metric', async () => {
// set process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT to true
process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'true';
const headers: ResponseHeaders = {
'x-ratelimit-remaining': '10',
Expand All @@ -59,7 +49,6 @@ describe('metricGitHubAppRateLimit', () => {
});

it('should not update rate limit metric', async () => {
// set process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT to false
process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'false';
const headers: ResponseHeaders = {
'x-ratelimit-remaining': '10',
Expand All @@ -72,103 +61,44 @@ describe('metricGitHubAppRateLimit', () => {
});

it('should not update rate limit metric if headers are undefined', async () => {
// set process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT to true
process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'true';

await metricGitHubAppRateLimit(undefined as unknown as ResponseHeaders);

expect(createSingleMetric).not.toHaveBeenCalled();
});

it('should cache GitHub App ID and only call getParameter once', async () => {
// Reset modules to clear the appIdPromises Map cache
vi.resetModules();
const { metricGitHubAppRateLimit: freshMetricFunction } = await import('./rate-limit');

it('should label metric with correct appId for index 1 (additional app)', async () => {
process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'true';
const headers: ResponseHeaders = {
'x-ratelimit-remaining': '10',
'x-ratelimit-limit': '60',
};

const mockGetParameter = vi.mocked(getParameter);
mockGetParameter.mockClear();

await freshMetricFunction(headers);
await freshMetricFunction(headers);
await freshMetricFunction(headers);

// getParameter should only be called once due to caching (index 0 cached after first call)
expect(mockGetParameter).toHaveBeenCalledTimes(1);
// split(':')[0] of 'test' is still 'test'
expect(mockGetParameter).toHaveBeenCalledWith(process.env.PARAMETER_GITHUB_APP_ID_NAME);
});
});

describe('metricGitHubAppRateLimit multi-app', () => {
let freshMetricFunction: typeof metricGitHubAppRateLimit;
let mockGetParam: ReturnType<typeof vi.fn>;

beforeEach(async () => {
// Reset modules to get a clean appIdPromises Map for each test
vi.resetModules();
const headers: ResponseHeaders = { 'x-ratelimit-remaining': '100', 'x-ratelimit-limit': '5000' };

process.env.PARAMETER_GITHUB_APP_ID_NAME = 'app0:app1';
process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'true';
await metricGitHubAppRateLimit(headers, 1);

mockGetParam = vi.fn((name: string) => {
if (name === 'app0') return Promise.resolve('1234');
if (name === 'app1') return Promise.resolve('5678');
return Promise.resolve('');
expect(createSingleMetric).toHaveBeenCalledWith('GitHubAppRateLimitRemaining', MetricUnit.Count, 100, {
AppId: '5678',
});

vi.doMock('@aws-github-runner/aws-ssm-util', () => ({ getParameter: mockGetParam }));
vi.doMock('@aws-github-runner/aws-powertools-util', () => ({
logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() },
createSingleMetric: vi.fn(() => ({ addMetadata: vi.fn() })),
}));

const mod = await import('./rate-limit');
freshMetricFunction = mod.metricGitHubAppRateLimit;
});

afterEach(() => {
vi.resetModules();
process.env.PARAMETER_GITHUB_APP_ID_NAME = 'test';
});
it('should default to index 0 when no appIndex is passed', async () => {
process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'true';
const headers: ResponseHeaders = { 'x-ratelimit-remaining': '75', 'x-ratelimit-limit': '5000' };

it('should label metric with correct appId for index 0 (primary app)', async () => {
const { createSingleMetric: mockMetric } = await import('@aws-github-runner/aws-powertools-util');
const headers: ResponseHeaders = { 'x-ratelimit-remaining': '50', 'x-ratelimit-limit': '5000' };
await freshMetricFunction(headers, 0);
expect(mockMetric).toHaveBeenCalledWith('GitHubAppRateLimitRemaining', MetricUnit.Count, 50, { AppId: '1234' });
});
await metricGitHubAppRateLimit(headers);

it('should label metric with correct appId for index 1 (additional app)', async () => {
const { createSingleMetric: mockMetric } = await import('@aws-github-runner/aws-powertools-util');
const headers: ResponseHeaders = { 'x-ratelimit-remaining': '100', 'x-ratelimit-limit': '5000' };
await freshMetricFunction(headers, 1);
expect(mockMetric).toHaveBeenCalledWith('GitHubAppRateLimitRemaining', MetricUnit.Count, 100, { AppId: '5678' });
expect(getLoadedAppId).toHaveBeenCalledWith(0);
expect(createSingleMetric).toHaveBeenCalledWith('GitHubAppRateLimitRemaining', MetricUnit.Count, 75, {
AppId: '1234',
});
});

it('should default to index 0 when no appIndex is passed', async () => {
const { createSingleMetric: mockMetric } = await import('@aws-github-runner/aws-powertools-util');
it('should label metric with an empty AppId when the appIndex is unknown', async () => {
process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'true';
const headers: ResponseHeaders = { 'x-ratelimit-remaining': '75', 'x-ratelimit-limit': '5000' };
await freshMetricFunction(headers);
expect(mockMetric).toHaveBeenCalledWith('GitHubAppRateLimitRemaining', MetricUnit.Count, 75, { AppId: '1234' });
});

it('should cache per index and call getParameter separately for each index', async () => {
const headers: ResponseHeaders = { 'x-ratelimit-remaining': '10', 'x-ratelimit-limit': '5000' };
await metricGitHubAppRateLimit(headers, 99);

// Two calls with index 1, then one with index 0
await freshMetricFunction(headers, 1);
await freshMetricFunction(headers, 1);
await freshMetricFunction(headers, 0);

// getParameter should be called exactly once per distinct index
expect(mockGetParam).toHaveBeenCalledTimes(2);
expect(mockGetParam).toHaveBeenCalledWith('app1');
expect(mockGetParam).toHaveBeenCalledWith('app0');
expect(createSingleMetric).toHaveBeenCalledWith('GitHubAppRateLimitRemaining', MetricUnit.Count, 75, {
AppId: '',
});
});
});
Loading
Loading