init
This commit is contained in:
redlineo committed
commit
d295c7f139
1527 files changed
+270726
No files matched your search
@@ -0,0 +1,117 @@
|
||||
# career-ops plugins
|
||||
|
||||
The plugin layer is the **opt-in home for integrations that need a key or talk
|
||||
to an external service** — things the zero-keys, local-first core deliberately
|
||||
doesn't carry. It generalizes the proven `providers/` pattern: drop a directory
|
||||
in here, declare a manifest, and it's discovered automatically.
|
||||
|
||||
> **Not the Claude Code plugin.** This is unrelated to `.claude-plugin/` (the
|
||||
> Claude Code marketplace metadata). These plugins extend career-ops itself.
|
||||
|
||||
## Default: off
|
||||
|
||||
Plugins load **only** when you opt in. With no `config/plugins.yml`, the core
|
||||
runs exactly as it always has — no plugin code runs, no `.env` is read, nothing
|
||||
changes. Two gates must both be satisfied:
|
||||
|
||||
1. **Enable** the plugin in `config/plugins.yml` (copy `config/plugins.example.yml`).
|
||||
2. **Provide its keys** in your own `.env` (each plugin declares which it needs).
|
||||
Run `node doctor.mjs` or `node plugins.mjs list` to see what's missing.
|
||||
|
||||
## Anatomy of a plugin
|
||||
|
||||
A plugin is a directory under `plugins/` (bundled, shipped with career-ops) or
|
||||
`plugins.local/` (your own, gitignored, never auto-updated):
|
||||
|
||||
```
|
||||
plugins/<id>/
|
||||
manifest.json # parsed, not executed — validated before any code is imported
|
||||
index.mjs # default-exports an object keyed by hook type
|
||||
_anything.mjs # helpers (the _ prefix means "never discovered as a plugin")
|
||||
```
|
||||
|
||||
### manifest.json
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "wellfound", // must equal the directory name; [a-z0-9-]
|
||||
"apiVersion": 1,
|
||||
"description": "One mission-framed line.",
|
||||
"hooks": ["provider"], // any of: provider, ingest, search, notify, export
|
||||
"requiredEnv": ["WELLFOUND_TOKEN"],// env VAR NAMES only — values go in .env
|
||||
"allowedHosts": ["api.wellfound.com"], // required when requiredEnv is non-empty
|
||||
"humanInTheLoop": true // must be true
|
||||
}
|
||||
```
|
||||
|
||||
### Hooks (`index.mjs` default export)
|
||||
|
||||
| Hook | Signature | Does |
|
||||
|------|-----------|------|
|
||||
| `provider` | `{ id, detect?, fetch(entry, ctx) → Job[] }` | A keyed/auth-gated job source. Same shape as `providers/_types.js`. Runs via `scan` on a `provider: <id>` entry in `portals.yml`. |
|
||||
| `ingest` | `(ctx) → Job[]` | Pull postings from a service (email, a board). |
|
||||
| `search` | `(query, ctx) → Job[]` | Postings for a query string. |
|
||||
| `export` | `(snapshot, ctx) → {pushed}` | Push a **read-only** tracker snapshot to your own external store. |
|
||||
| `notify` | `(payload, ctx) → void` | Send an outbound notification. |
|
||||
|
||||
Producers (`provider`/`ingest`/`search`) **return** `Job[]`
|
||||
(`{title, url, company, location}`); the engine — never the plugin — writes them
|
||||
to `data/pipeline.md` through the canonical writer, so a plugin can't break the
|
||||
data formats the web reads. Non-provider hooks run explicitly:
|
||||
|
||||
```bash
|
||||
node plugins.mjs list
|
||||
node plugins.mjs run gmail # ingest
|
||||
node plugins.mjs run notion search "platform" # search
|
||||
node plugins.mjs run notion export [--dry-run] # export
|
||||
```
|
||||
|
||||
### The `ctx` object
|
||||
|
||||
- `fetch(url, opts)` — the **guarded** primitive: HTTPS-only, pinned to your
|
||||
`allowedHosts`, `redirect:'manual'` re-validating every hop and stripping
|
||||
credentials on a hostname change. **Route your HTTP through `ctx.fetch`** (or
|
||||
the `fetchText`/`fetchJson` conveniences over it) so the egress guard actually
|
||||
runs — a plugin that calls global `fetch` bypasses it (the bundled `apify`
|
||||
plugin is one deliberate exception: its client self-constrains to a single
|
||||
hardcoded host, documented in its code).
|
||||
- `env` (frozen, scoped to your declared keys), `settings` (your non-secret
|
||||
`config/plugins.yml` block), `log` (redacts your declared secrets), `dryRun`.
|
||||
|
||||
## Your own plugins → `plugins.local/`
|
||||
|
||||
Put private or experimental plugins in **`plugins.local/`** (a sibling of
|
||||
`plugins/`), never in `plugins/`. `plugins.local/` is gitignored and never
|
||||
auto-updated, so updates can't clobber it and a same-id bundled plugin can't be
|
||||
shadowed by it. Bundled plugins always win an id collision.
|
||||
|
||||
## Trust model (read this)
|
||||
|
||||
career-ops is plain ESM with no build step, so the engine **cannot truly
|
||||
sandbox** a plugin's imports. `allowedHosts`, the scoped `ctx.env`, and the
|
||||
no-auto-submit hook taxonomy constrain an **honest** plugin and make every loaded
|
||||
plugin visible (`doctor` / `plugins.mjs list`) — but they are not a hard
|
||||
boundary against malicious code, which can reach `process.env` or the network
|
||||
directly. Containment is the same as everywhere else in open source:
|
||||
|
||||
- **Bundled plugins** (`plugins/`) are code-reviewed exactly like `providers/`.
|
||||
CI checks that they declare no core-owned secret, import no browser-automation
|
||||
or process-spawning module, and never auto-submit.
|
||||
- **`plugins.local/`** runs with **your** trust — you installed it. Treat a
|
||||
third-party plugin like any code you run on your machine.
|
||||
|
||||
## Not a plugin
|
||||
|
||||
These don't belong in the plugin layer — they're a different direction:
|
||||
|
||||
- **Centralized infrastructure** the project would operate — hosted job
|
||||
aggregation, a shared matching service, proxies/Workers. That's a **separate,
|
||||
opt-in service**, discussed in
|
||||
[Where career-ops is going (#904)](https://github.com/santifer/career-ops/discussions/904) —
|
||||
not the open-core.
|
||||
- **Auto-submitting / blind-applying** to jobs. career-ops is a decision-support
|
||||
tool, not a spam bot — it drafts applications for **you** to review and submit.
|
||||
No hook can submit, and `humanInTheLoop: true` is mandatory. This holds
|
||||
everywhere, in core and plugins alike.
|
||||
|
||||
See `CONTRIBUTING.md` → "Scope" for the full boundary.
|
||||
@@ -0,0 +1,720 @@
|
||||
// @ts-check
|
||||
/**
|
||||
* plugins/_engine.mjs — the career-ops plugin engine.
|
||||
*
|
||||
* Generalizes the proven providers/ auto-loader (scan.mjs `loadProviders`) into
|
||||
* a sibling `plugins/` layer for integrations that need a KEY or talk to an
|
||||
* EXTERNAL service. The zero-key providers/ dir is untouched and stays pure.
|
||||
*
|
||||
* Design invariants (every one is asserted by test-all.mjs section 49):
|
||||
* - ZERO module-level side effects. Importing this file reads no config, loads
|
||||
* no dotenv, and mutates no process.env. scan.mjs imports `mergeProviderPlugins`
|
||||
* on every run, and that import must be free — so a plain `node scan.mjs` with
|
||||
* no config/plugins.yml behaves byte-identically to before this feature.
|
||||
* - Fail-open everywhere. A malformed manifest, a throwing import, a missing
|
||||
* key, or a hung hook logs a `⚠️` and is SKIPPED — never crashes the core.
|
||||
* - Opt-in. Nothing loads unless config/plugins.yml enables it AND every
|
||||
* declared requiredEnv var is present.
|
||||
* - Least-privilege ctx is a CONVENIENCE, not a sandbox. Plain ESM can't
|
||||
* VM-isolate a module's imports without a build step (forbidden). Containment
|
||||
* is code review (bundled plugins, same gate as providers/) + user trust
|
||||
* (plugins.local/). See plugins/README.md "Trust model".
|
||||
*
|
||||
* Pure + side-effect-free so scan.mjs, plugins.mjs, doctor.mjs and test-all.mjs
|
||||
* all reuse it with no prod-vs-test drift.
|
||||
*/
|
||||
|
||||
import { existsSync, readdirSync, readFileSync, realpathSync } from 'fs';
|
||||
import path from 'path';
|
||||
import { pathToFileURL } from 'url';
|
||||
import { resolveAndValidate } from './_net.mjs';
|
||||
import { readLock, writeLockEntry, diffPlugin, hashPluginTree, consentSurface } from './_lock.mjs';
|
||||
import { loadRegistry } from './_registry.mjs';
|
||||
|
||||
/** The complete, closed set of hook kinds. Anything else (apply/submit/…) is rejected. */
|
||||
export const HOOK_KINDS = ['provider', 'ingest', 'search', 'notify', 'export'];
|
||||
|
||||
/**
|
||||
* Env var names a plugin may NOT declare in requiredEnv/optionalEnv: core-owned
|
||||
* secrets (so a plugin can't smuggle out GEMINI/OPENAI/… keys and have doctor
|
||||
* render it as a legitimate ✓), plus process-shaping vars. AWS_* is matched by
|
||||
* prefix below. This is a manifest-review backstop, not a runtime isolation
|
||||
* boundary (process.env stays globally reachable — see the trust note).
|
||||
*/
|
||||
export const RESERVED_ENV = new Set([
|
||||
'GEMINI_API_KEY', 'GEMINI_MODEL',
|
||||
'OPENROUTER_API_KEY', 'CAREER_OPS_MODEL',
|
||||
'OPENAI_API_KEY', 'OPENAI_BASE_URL', 'OPENAI_MODEL',
|
||||
'ANTHROPIC_API_KEY',
|
||||
'CAREER_OPS_PORTALS', 'CAREER_OPS_PROFILE',
|
||||
'PATH', 'HOME', 'NODE_OPTIONS', 'LD_PRELOAD', 'NODE_EXTRA_CA_CERTS',
|
||||
]);
|
||||
|
||||
const ID_RE = /^[a-z0-9][a-z0-9-]*$/;
|
||||
const DEFAULT_HOOK_TIMEOUT_MS = 15_000;
|
||||
const MAX_REDIRECTS = 5;
|
||||
|
||||
function isReservedEnv(name) {
|
||||
return RESERVED_ENV.has(name) || /^AWS_/.test(name);
|
||||
}
|
||||
|
||||
function warnSkip(label, reason) {
|
||||
console.warn(`⚠️ ${label}: skipping — ${reason}`);
|
||||
}
|
||||
|
||||
function isWithinDirectory(rootAbs, candidateAbs) {
|
||||
const rel = path.relative(rootAbs, candidateAbs);
|
||||
return rel === '' || (!rel.startsWith(`..${path.sep}`) && rel !== '..' && !path.isAbsolute(rel));
|
||||
}
|
||||
|
||||
function nearestExistingPath(absPath) {
|
||||
let current = path.resolve(absPath);
|
||||
while (!existsSync(current)) {
|
||||
const parent = path.dirname(current);
|
||||
if (parent === current) return null;
|
||||
current = parent;
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
function isSafePluginPath(rootAbs, candidateAbs) {
|
||||
const rootResolved = path.resolve(rootAbs);
|
||||
const candidateResolved = path.resolve(candidateAbs);
|
||||
if (!isWithinDirectory(rootResolved, candidateResolved)) return false;
|
||||
|
||||
const nearestExisting = nearestExistingPath(candidateResolved);
|
||||
if (!nearestExisting) return false;
|
||||
try {
|
||||
return isWithinDirectory(realpathSync(rootResolved), realpathSync(nearestExisting));
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the config/plugins.yml path for a given project root.
|
||||
* @param {string} root
|
||||
*/
|
||||
function pluginsConfigPath(root) {
|
||||
return path.join(root, 'config', 'plugins.yml');
|
||||
}
|
||||
|
||||
/**
|
||||
* Read config/plugins.yml. Fail-open to {} if absent or malformed — exactly the
|
||||
* graceful posture scan.mjs uses for its own config. js-yaml is imported lazily
|
||||
* so this module stays side-effect-free at import time.
|
||||
* @param {string} root
|
||||
* @returns {Promise<object>}
|
||||
*/
|
||||
export async function loadPluginConfig(root) {
|
||||
const file = pluginsConfigPath(root);
|
||||
if (!existsSync(file)) return {};
|
||||
try {
|
||||
const yaml = (await import('js-yaml')).default;
|
||||
const parsed = yaml.load(readFileSync(file, 'utf8'));
|
||||
return parsed && typeof parsed === 'object' ? parsed : {};
|
||||
} catch (err) {
|
||||
warnSkip('config/plugins.yml', `unreadable, ignoring — ${err.message}`);
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a parsed manifest object. Returns a normalized manifest on success,
|
||||
* or null (with a ⚠️) on any violation. NEVER throws — identical fail-open to
|
||||
* scan.mjs loadProviders.
|
||||
*
|
||||
* @param {any} m Parsed manifest.json.
|
||||
* @param {string} dir Absolute plugin directory.
|
||||
* @param {string} dirName Basename of dir (must equal m.id).
|
||||
* @returns {(PluginManifestNormalized|null)}
|
||||
* @typedef {object} PluginManifestNormalized
|
||||
* @property {string} id
|
||||
* @property {number} apiVersion
|
||||
* @property {string} description
|
||||
* @property {string[]} hooks
|
||||
* @property {string[]} requiredEnv
|
||||
* @property {string[]} optionalEnv
|
||||
* @property {string[]} allowedHosts
|
||||
* @property {string} entry
|
||||
* @property {boolean} humanInTheLoop
|
||||
* @property {string} [name]
|
||||
* @property {string} [version]
|
||||
* @property {string} [homepage]
|
||||
* @property {string} dir
|
||||
*/
|
||||
export function validateManifest(m, dir, dirName) {
|
||||
const label = dirName;
|
||||
if (!m || typeof m !== 'object') { warnSkip(label, 'manifest.json is not an object'); return null; }
|
||||
|
||||
if (typeof m.id !== 'string' || !ID_RE.test(m.id)) { warnSkip(label, `invalid id ${JSON.stringify(m.id)} (need ^[a-z0-9][a-z0-9-]*$)`); return null; }
|
||||
if (m.id !== dirName) { warnSkip(label, `id "${m.id}" must equal directory name "${dirName}"`); return null; }
|
||||
if (m.apiVersion !== 1) { warnSkip(label, `unsupported apiVersion ${JSON.stringify(m.apiVersion)} (engine supports 1)`); return null; }
|
||||
if (typeof m.description !== 'string' || !m.description.trim() || /[\r\n]/.test(m.description)) { warnSkip(label, 'description must be a non-empty single line'); return null; }
|
||||
if (m.humanInTheLoop !== true) { warnSkip(label, 'humanInTheLoop must be true (no auto-submit plugins)'); return null; }
|
||||
|
||||
if (!Array.isArray(m.hooks) || m.hooks.length === 0) { warnSkip(label, 'hooks must be a non-empty array'); return null; }
|
||||
for (const h of m.hooks) {
|
||||
if (!HOOK_KINDS.includes(h)) { warnSkip(label, `unknown hook "${h}" (allowed: ${HOOK_KINDS.join(', ')})`); return null; }
|
||||
}
|
||||
|
||||
const requiredEnv = m.requiredEnv === undefined ? [] : m.requiredEnv;
|
||||
const optionalEnv = m.optionalEnv === undefined ? [] : m.optionalEnv;
|
||||
if (!Array.isArray(requiredEnv) || requiredEnv.some(x => typeof x !== 'string')) { warnSkip(label, 'requiredEnv must be an array of strings'); return null; }
|
||||
if (!Array.isArray(optionalEnv) || optionalEnv.some(x => typeof x !== 'string')) { warnSkip(label, 'optionalEnv must be an array of strings'); return null; }
|
||||
for (const name of [...requiredEnv, ...optionalEnv]) {
|
||||
if (isReservedEnv(name)) { warnSkip(label, `requiredEnv/optionalEnv may not declare reserved var "${name}" (core-owned or process-shaping)`); return null; }
|
||||
}
|
||||
|
||||
const allowedHosts = m.allowedHosts === undefined ? [] : m.allowedHosts;
|
||||
if (!Array.isArray(allowedHosts) || allowedHosts.some(x => typeof x !== 'string')) { warnSkip(label, 'allowedHosts must be an array of strings'); return null; }
|
||||
// A keyed plugin (reads a secret) MUST declare an egress allowlist.
|
||||
if (requiredEnv.length > 0 && allowedHosts.length === 0) { warnSkip(label, 'a plugin with requiredEnv must declare a non-empty allowedHosts egress allowlist'); return null; }
|
||||
|
||||
const allowsLocalhost = m.allowsLocalhost === true;
|
||||
if (allowsLocalhost && allowedHosts.length === 0) { warnSkip(label, 'allowsLocalhost requires a non-empty allowedHosts'); return null; }
|
||||
// Reject SSRF-shaped hosts in the allowlist (IP literals, cloud-metadata,
|
||||
// *.internal, loopback) unless the plugin explicitly opted into localhost.
|
||||
for (const h of allowedHosts) {
|
||||
const isLoopback = /^(localhost|127\.\d+\.\d+\.\d+|::1|\[::1\])$/i.test(h);
|
||||
if (/^\d{1,3}(\.\d{1,3}){3}$/.test(h) && !(allowsLocalhost && isLoopback)) { warnSkip(label, `allowedHosts must be hostnames, not IP literals: ${h}`); return null; }
|
||||
if (/(^169\.254\.)|(^metadata\.google\.internal$)|(\.internal$)/i.test(h)) { warnSkip(label, `allowedHosts may not target metadata/internal hosts: ${h}`); return null; }
|
||||
if (isLoopback && !allowsLocalhost) { warnSkip(label, `allowedHosts contains a loopback host without allowsLocalhost: ${h}`); return null; }
|
||||
}
|
||||
|
||||
const entry = typeof m.entry === 'string' && m.entry.trim() ? m.entry : 'index.mjs';
|
||||
if (!entry.endsWith('.mjs')) { warnSkip(label, `entry "${entry}" must be a .mjs file`); return null; }
|
||||
const entryAbs = path.resolve(dir, entry);
|
||||
if (!isSafePluginPath(dir, entryAbs)) { warnSkip(label, `entry "${entry}" escapes the plugin directory`); return null; }
|
||||
|
||||
// Optional companion skill (Open-Agent-Skill-Standard SKILL.md). Traversal-
|
||||
// guarded like entry. Surfaced on-demand via `plugins.mjs skill <id>`; never
|
||||
// auto-injected into AGENTS.md/modes (the data-vs-brain firewall).
|
||||
let skill = null;
|
||||
if (m.skill !== undefined) {
|
||||
if (typeof m.skill !== 'string' || !m.skill.endsWith('.md')) { warnSkip(label, 'skill must be a relative .md path'); return null; }
|
||||
const skillAbs = path.resolve(dir, m.skill);
|
||||
if (!isSafePluginPath(dir, skillAbs)) { warnSkip(label, `skill "${m.skill}" escapes the plugin directory`); return null; }
|
||||
if (!existsSync(skillAbs)) { warnSkip(label, `skill file not found: ${m.skill}`); return null; }
|
||||
skill = m.skill;
|
||||
}
|
||||
|
||||
return {
|
||||
id: m.id,
|
||||
apiVersion: 1,
|
||||
description: m.description.trim(),
|
||||
hooks: [...m.hooks],
|
||||
requiredEnv,
|
||||
optionalEnv,
|
||||
allowedHosts,
|
||||
allowsLocalhost,
|
||||
entry,
|
||||
skill,
|
||||
humanInTheLoop: true,
|
||||
name: typeof m.name === 'string' ? m.name : undefined,
|
||||
version: typeof m.version === 'string' ? m.version : undefined,
|
||||
homepage: typeof m.homepage === 'string' ? m.homepage : undefined,
|
||||
dir,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Discover plugin directories across roots. Roots are scanned in order; the
|
||||
* FIRST occurrence of an id wins (so bundled plugins/ shadow a same-id
|
||||
* plugins.local/). existsSync-guarded (never mkdir — a fresh, plugin-free
|
||||
* install stays filesystem-inert). Alphabetical within a root for determinism.
|
||||
*
|
||||
* `overrideIds` is the ONE controlled exception to first-root-wins: an id in
|
||||
* this set lets a LATER root (plugins.local/) override an earlier one
|
||||
* (bundled plugins/). It is computed ONLY by resolveSuccessorIds() — i.e. a
|
||||
* community plugin the maintainer registered as the bundled plugin's
|
||||
* `supersedesBundled` successor AND that the user installed at its exact
|
||||
* pinned sha. A plain, unverified plugins.local/<id> is never in the set, so
|
||||
* it can NEVER shadow a reviewed bundled plugin (the no-downgrade invariant).
|
||||
* Default empty → identical to the original bundled-always-wins behavior.
|
||||
*
|
||||
* @param {string[]} roots Absolute directories to scan (e.g. [plugins/, plugins.local/]).
|
||||
* @param {Set<string>} [overrideIds] ids whose later-root copy may override an earlier-root one.
|
||||
* @returns {PluginManifestNormalized[]}
|
||||
*/
|
||||
export function discoverPlugins(roots, overrideIds = new Set()) {
|
||||
const found = new Map();
|
||||
for (const root of roots) {
|
||||
if (!existsSync(root)) continue;
|
||||
let entries;
|
||||
try {
|
||||
entries = readdirSync(root, { withFileTypes: true });
|
||||
} catch (err) {
|
||||
warnSkip(root, `unreadable — ${err.message}`);
|
||||
continue;
|
||||
}
|
||||
const dirs = entries
|
||||
.filter(e => e.isDirectory() && !e.name.startsWith('_') && !e.name.startsWith('.'))
|
||||
.map(e => e.name)
|
||||
.sort();
|
||||
for (const name of dirs) {
|
||||
const dir = path.join(root, name);
|
||||
const manifestFile = path.join(dir, 'manifest.json');
|
||||
if (!existsSync(manifestFile)) continue; // not a plugin dir (no manifest)
|
||||
let parsed;
|
||||
try {
|
||||
parsed = JSON.parse(readFileSync(manifestFile, 'utf8'));
|
||||
} catch (err) {
|
||||
warnSkip(name, `manifest.json is invalid JSON — ${err.message}`);
|
||||
continue;
|
||||
}
|
||||
const manifest = validateManifest(parsed, dir, name);
|
||||
if (!manifest) continue;
|
||||
if (found.has(manifest.id)) {
|
||||
// earlier root wins (bundled > local) — EXCEPT an approved successor.
|
||||
if (overrideIds.has(manifest.id)) found.set(manifest.id, manifest);
|
||||
continue;
|
||||
}
|
||||
found.set(manifest.id, manifest);
|
||||
}
|
||||
}
|
||||
return [...found.values()];
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the set of bundled ids that an APPROVED community successor should
|
||||
* override. A successor qualifies only when ALL hold:
|
||||
* (a) a registry entry declares `supersedesBundled: true` for that id
|
||||
* (registry = SYSTEM file → a maintainer reviewed + merged it), AND
|
||||
* (b) plugins.local/<id> is actually installed, AND
|
||||
* (c) the installed sha === the registry's pinned sha (the user has the
|
||||
* exact reviewed commit — the same bar classifySource() calls 'approved').
|
||||
* Anything short of (c) — off-registry drift, unverified, not installed —
|
||||
* leaves the bundled reference in charge. This is the trust hinge of the whole
|
||||
* seed/successor model: only a reviewed, pinned, installed successor wins.
|
||||
* @param {string} root
|
||||
* @returns {Set<string>}
|
||||
*/
|
||||
export function resolveSuccessorIds(root) {
|
||||
const ids = new Set();
|
||||
try {
|
||||
const reg = loadRegistry(root);
|
||||
const lock = readLock(root);
|
||||
for (const e of reg.plugins) {
|
||||
if (e.supersedesBundled !== true || typeof e.id !== 'string') continue;
|
||||
const localManifest = path.join(root, 'plugins.local', e.id, 'manifest.json');
|
||||
const installedSha = lock?.plugins?.[e.id]?.sha;
|
||||
if (existsSync(localManifest) && installedSha && installedSha === e.sha) ids.add(e.id);
|
||||
}
|
||||
} catch {
|
||||
// Any registry/lock read problem → no overrides (bundled references stay in charge).
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
/**
|
||||
* The standard pair of roots for a project: bundled plugins/ then user
|
||||
* plugins.local/ (the latter only matters when it exists).
|
||||
* @param {string} root
|
||||
*/
|
||||
export function pluginRoots(root) {
|
||||
return [path.join(root, 'plugins'), path.join(root, 'plugins.local')];
|
||||
}
|
||||
|
||||
/**
|
||||
* Is a plugin enabled? Requires config/plugins.yml { plugins: { <id>: { enabled: true } } }
|
||||
* AND every requiredEnv var present in process.env.
|
||||
* @param {PluginManifestNormalized} manifest
|
||||
* @param {object} cfg Parsed config/plugins.yml.
|
||||
* @returns {{ enabled: boolean, configured: boolean, missingEnv: string[] }}
|
||||
*/
|
||||
export function pluginStatus(manifest, cfg) {
|
||||
const entry = cfg?.plugins?.[manifest.id];
|
||||
const configured = entry?.enabled === true;
|
||||
const missingEnv = manifest.requiredEnv.filter(name => !process.env[name]);
|
||||
return { enabled: configured && missingEnv.length === 0, configured, missingEnv };
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-plugin non-secret settings block from config/plugins.yml (e.g.
|
||||
* { actor: "...", database_id: "..." }). Secrets never live here.
|
||||
* @param {string} id
|
||||
* @param {object} cfg
|
||||
*/
|
||||
export function pluginSettings(id, cfg) {
|
||||
const entry = cfg?.plugins?.[id];
|
||||
if (!entry || typeof entry !== 'object') return {};
|
||||
const { enabled, ...rest } = entry;
|
||||
return rest;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a guarded fetch that enforces HTTPS, an optional host allowlist, and
|
||||
* manual redirect handling that re-validates EVERY hop's host and strips
|
||||
* credential headers when a hop changes hostname. Returns a Response.
|
||||
*
|
||||
* Posture note: core providers use redirect:'error' (reject ANY redirect). This
|
||||
* is the deliberately looser plugin posture — allowlist-pinned FOLLOW — because
|
||||
* keyed APIs (Notion/Google/Apify) legitimately 30x within their own host set;
|
||||
* the allowlist + per-hop re-validation + cross-host credential strip bound it.
|
||||
*
|
||||
* ADVISORY only: this binds a plugin that routes through ctx.fetch*, not one
|
||||
* that calls global fetch directly (see the trust note in README.md).
|
||||
*
|
||||
* @param {string[]} allowedHosts
|
||||
*/
|
||||
function makeGuardedFetch(allowedHosts, { allowsLocalhost = false } = {}) {
|
||||
const allow = new Set(allowedHosts);
|
||||
const isLoopbackHost = (h) => /^(localhost|127\.\d+\.\d+\.\d+|\[?::1\]?)$/i.test(h);
|
||||
const hostOk = (u) => {
|
||||
if (u.protocol !== 'https:') {
|
||||
// Plain HTTP is allowed ONLY for an opted-in loopback host (local-AI
|
||||
// providers like Ollama/LM Studio serve http://localhost:11434).
|
||||
if (!(allowsLocalhost && u.protocol === 'http:' && isLoopbackHost(u.hostname))) {
|
||||
throw new Error(`plugin egress must use HTTPS: ${u.href}`);
|
||||
}
|
||||
}
|
||||
if (allow.size > 0 && !allow.has(u.hostname)) throw new Error(`plugin egress to "${u.hostname}" is not in allowedHosts [${[...allow].join(', ')}]`);
|
||||
};
|
||||
return async function guardedFetch(url, opts = {}) {
|
||||
const { timeoutMs = 10_000, headers = {}, method = 'GET', body = null } = opts;
|
||||
let current = new URL(url);
|
||||
hostOk(current);
|
||||
// SSRF: reject a host that resolves to a private/loopback/metadata address
|
||||
// (re-checked on every redirect hop). Loopback allowed only when opted in.
|
||||
await resolveAndValidate(current.hostname, { allowsLocalhost });
|
||||
let reqHeaders = { ...headers };
|
||||
for (let hop = 0; hop <= MAX_REDIRECTS; hop++) {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
||||
let res;
|
||||
try {
|
||||
res = await fetch(current.href, {
|
||||
method, headers: reqHeaders, body, redirect: 'manual', signal: controller.signal,
|
||||
});
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
if (res.status >= 300 && res.status < 400 && res.headers.get('location')) {
|
||||
const next = new URL(res.headers.get('location'), current);
|
||||
hostOk(next); // re-validate the redirect target before following it
|
||||
await resolveAndValidate(next.hostname, { allowsLocalhost });
|
||||
if (next.hostname !== current.hostname) {
|
||||
// Don't forward credentials across a hostname change (what the
|
||||
// platform fetch does for cross-origin redirects; we do it manually).
|
||||
reqHeaders = Object.fromEntries(Object.entries(reqHeaders).filter(([k]) => !/^(authorization|cookie)$/i.test(k)));
|
||||
}
|
||||
current = next;
|
||||
continue;
|
||||
}
|
||||
if (!res.ok) {
|
||||
const snippet = (await res.text().catch(() => '')).replace(/\s+/g, ' ').trim().slice(0, 300);
|
||||
const err = new Error(snippet ? `HTTP ${res.status}: ${snippet}` : `HTTP ${res.status}`);
|
||||
// @ts-ignore
|
||||
err.status = res.status;
|
||||
throw err;
|
||||
}
|
||||
return res;
|
||||
}
|
||||
throw new Error(`too many redirects (>${MAX_REDIRECTS}) for ${url}`);
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the least-privilege ctx for a plugin. The scoped frozen env is a
|
||||
* CONVENIENCE (process.env is still globally reachable from any module) — the
|
||||
* real boundary is code review + trust.
|
||||
* @param {PluginManifestNormalized} manifest
|
||||
* @param {{ dryRun?: boolean, settings?: object }} [opts]
|
||||
* @returns {PluginContext}
|
||||
*/
|
||||
export function buildCtx(manifest, opts = {}) {
|
||||
const scoped = {};
|
||||
for (const name of [...manifest.requiredEnv, ...manifest.optionalEnv]) {
|
||||
if (process.env[name] !== undefined) scoped[name] = process.env[name];
|
||||
}
|
||||
const env = Object.freeze({ ...scoped });
|
||||
// Secret values long enough to be worth redacting (avoid no-op/over-redaction
|
||||
// on empty or trivially short strings).
|
||||
const secrets = manifest.requiredEnv
|
||||
.map(n => process.env[n])
|
||||
.filter(v => typeof v === 'string' && v.length >= 6);
|
||||
const guarded = makeGuardedFetch(manifest.allowedHosts, { allowsLocalhost: manifest.allowsLocalhost === true });
|
||||
const log = (...args) => {
|
||||
const redact = (s) => {
|
||||
let out = typeof s === 'string' ? s : String(s);
|
||||
for (const sec of secrets) out = out.split(sec).join('«redacted»');
|
||||
return out;
|
||||
};
|
||||
console.log(...args.map(redact));
|
||||
};
|
||||
return /** @type {PluginContext} */ ({
|
||||
transport: 'http',
|
||||
// The guarded primitive (HTTPS + allowedHosts + redirect re-validation).
|
||||
// Bundled plugins should route their HTTP through this so the egress guard
|
||||
// actually runs; fetchText/fetchJson are conveniences over it.
|
||||
fetch: guarded,
|
||||
fetchText: async (u, o) => (await guarded(u, o)).text(),
|
||||
fetchJson: async (u, o) => (await guarded(u, o)).json(),
|
||||
env,
|
||||
settings: Object.freeze({ ...(opts.settings || {}) }),
|
||||
log,
|
||||
dryRun: opts.dryRun === true,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Dynamically import a plugin entry and return its validated hook object for the
|
||||
* requested kind, or null on any failure (sandboxed — never throws to caller).
|
||||
* @param {PluginManifestNormalized} manifest
|
||||
* @param {string} kind
|
||||
*/
|
||||
async function importHook(manifest, kind) {
|
||||
const entryAbs = path.join(manifest.dir, manifest.entry);
|
||||
let mod;
|
||||
try {
|
||||
mod = await import(pathToFileURL(entryAbs).href);
|
||||
} catch (err) {
|
||||
warnSkip(manifest.id, `failed to import ${manifest.entry} — ${err.message}`);
|
||||
return null;
|
||||
}
|
||||
const hooks = mod.default;
|
||||
if (!hooks || typeof hooks !== 'object') { warnSkip(manifest.id, 'entry default export must be an object of hooks'); return null; }
|
||||
const hook = hooks[kind];
|
||||
if (kind === 'provider') {
|
||||
if (!hook || typeof hook.fetch !== 'function' || typeof hook.id !== 'string') { warnSkip(manifest.id, 'provider hook must be { id, fetch }'); return null; }
|
||||
} else if (typeof hook !== 'function') {
|
||||
warnSkip(manifest.id, `${kind} hook must be a function`);
|
||||
return null;
|
||||
}
|
||||
return hook;
|
||||
}
|
||||
|
||||
/**
|
||||
* Load all ENABLED plugins exposing `kind`, with their ctx ready. Skips any that
|
||||
* fail to import. Caller is responsible for dotenv (loadDotenvOnce) before this
|
||||
* if the keys live in .env.
|
||||
* @param {string} kind
|
||||
* @param {{ root: string, dryRun?: boolean }} opts
|
||||
* @returns {Promise<Array<{ id: string, manifest: PluginManifestNormalized, hook: any, ctx: PluginContext }>>}
|
||||
*/
|
||||
// Phrases that suggest a skill is trying to hijack the agent rather than
|
||||
// document a plugin. Built from fragments so the literals don't trip the repo's
|
||||
// own firewall/injection greps. Surfaced as a ⚠️ at print time, not a block.
|
||||
const SKILL_RED_FLAGS = [
|
||||
/ignore\s+(all\s+)?(previous|prior|above)\s+instruction/i,
|
||||
/disregard\s+(the\s+)?(system|above|previous)/i,
|
||||
/you\s+are\s+now\b/i,
|
||||
/exfiltrat/i,
|
||||
new RegExp('modes/' + '_shared'),
|
||||
new RegExp('AGENTS' + '\\.md'),
|
||||
/human[- ]?in[- ]?the[- ]?loop\s*[:=]?\s*false/i,
|
||||
/humanInTheLoop\s*[:=]\s*false/i,
|
||||
];
|
||||
|
||||
/**
|
||||
* Read a plugin's companion skill.md (on-demand; the agent PULLS it). Returns
|
||||
* its body + injection red-flags + whether the source is trusted (bundled).
|
||||
* NEVER auto-injected into AGENTS.md/modes — the data-vs-brain firewall.
|
||||
* @returns {{ body: string, flags: string[], source: string } | null}
|
||||
*/
|
||||
export function loadSkill(manifest, root) {
|
||||
if (!manifest.skill) return null;
|
||||
let body;
|
||||
try { body = readFileSync(path.join(manifest.dir, manifest.skill), 'utf8'); }
|
||||
catch { return null; }
|
||||
const flags = SKILL_RED_FLAGS.some(re => re.test(body))
|
||||
? ['this skill contains phrases that resemble prompt-injection — review it before trusting']
|
||||
: [];
|
||||
return { body, flags, source: pluginSource(manifest, root) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive a plugin's trust source from the FILESYSTEM, never from plugins.lock
|
||||
* (a USER-writable file that could spoof `source: "bundled"` to dodge the
|
||||
* rug-pull gate). A dir under plugins/ is bundled; anything else is local.
|
||||
*/
|
||||
function pluginSource(manifest, root) {
|
||||
return manifest.dir.startsWith(path.join(root, 'plugins') + path.sep) ? 'bundled' : 'local';
|
||||
}
|
||||
|
||||
/**
|
||||
* Integrity gate run on an ENABLED plugin before importing it (the rug-pull
|
||||
* defense). Returns { load }. Re-pins the lock on the benign cases (silent).
|
||||
* Fail-open on the engine itself (a lock-read error never blocks).
|
||||
*/
|
||||
export function lockGate(manifest, root) {
|
||||
const source = pluginSource(manifest, root);
|
||||
let lock;
|
||||
try { lock = readLock(root); } catch { return { load: true }; }
|
||||
const entry = lock.plugins?.[manifest.id];
|
||||
let d;
|
||||
try { d = diffPlugin(manifest, entry); }
|
||||
catch (err) { warnSkip(manifest.id, `integrity check failed — ${err.message}`); return { load: false }; }
|
||||
|
||||
const repin = () => {
|
||||
try {
|
||||
const tree = hashPluginTree(manifest.dir);
|
||||
writeLockEntry(root, manifest.id, {
|
||||
source, version: manifest.version || '0.0.0',
|
||||
integrity: tree.integrity, files: tree.files, consent: consentSurface(manifest),
|
||||
});
|
||||
} catch { /* best-effort; never block load on a write failure */ }
|
||||
};
|
||||
|
||||
switch (d.status) {
|
||||
case 'unpinned': repin(); return { load: true }; // grandfathered hand-enable → pin so future tampering is caught
|
||||
case 'match': return { load: true };
|
||||
case 'legit-update': repin(); return { load: true }; // version bumped → honest update, re-pin quietly
|
||||
case 'drift-nobump':
|
||||
if (source === 'bundled') { repin(); return { load: true }; } // reviewed-by-construction (branch-protected checkout) → re-pin
|
||||
warnSkip(manifest.id, `files changed since you trusted it without a version bump — possible tampering (${d.changedFiles.slice(0, 5).join(', ')}). Review, then \`node plugins.mjs trust ${manifest.id}\``);
|
||||
return { load: false };
|
||||
case 'surface-widened':
|
||||
warnSkip(manifest.id, `capability surface expanded since you consented (${[...d.addedHosts, ...d.addedEnv].join(', ')}${manifest.allowsLocalhost ? ', localhost' : ''}) — re-consent: \`node plugins.mjs enable ${manifest.id}\``);
|
||||
return { load: false };
|
||||
default: return { load: true };
|
||||
}
|
||||
}
|
||||
|
||||
export async function loadPlugins(kind, { root, dryRun = false }) {
|
||||
const cfg = await loadPluginConfig(root);
|
||||
const manifests = discoverPlugins(pluginRoots(root), resolveSuccessorIds(root)).filter(m => m.hooks.includes(kind));
|
||||
const out = [];
|
||||
for (const manifest of manifests) {
|
||||
if (!pluginStatus(manifest, cfg).enabled) continue;
|
||||
if (!lockGate(manifest, root).load) continue;
|
||||
const hook = await importHook(manifest, kind);
|
||||
if (!hook) continue;
|
||||
out.push({ id: manifest.id, manifest, hook, ctx: buildCtx(manifest, { dryRun, settings: pluginSettings(manifest.id, cfg) }) });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Lazily load dotenv exactly once (mirrors gemini-eval.mjs). Idempotent. */
|
||||
let dotenvLoaded = false;
|
||||
export async function loadDotenvOnce() {
|
||||
if (dotenvLoaded) return;
|
||||
dotenvLoaded = true;
|
||||
try {
|
||||
const { config } = await import('dotenv');
|
||||
config();
|
||||
} catch {
|
||||
// dotenv optional — fall back to ambient process.env (CI, exported vars).
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one hook across all enabled plugins, each call try/caught + timed out.
|
||||
* Cooperative timeout only: Promise.race resolves the wait but does NOT abort a
|
||||
* synchronous-hung or process.exit-ing plugin (plain ESM can't preempt without a
|
||||
* worker — stated plainly in README). One throwing/slow plugin can't sink the
|
||||
* batch; a sync-hang or a process.exit can. Bundled plugins are reviewed to
|
||||
* avoid both, exactly like providers/.
|
||||
*
|
||||
* @param {string} kind
|
||||
* @param {*} payload For provider this is unused; for ingest none; search a query; export a snapshot; notify a payload.
|
||||
* @param {{ root: string, dryRun?: boolean, timeoutMs?: number }} opts
|
||||
* @returns {Promise<Array<{ id: string, ok: boolean, result?: any, error?: string }>>}
|
||||
*/
|
||||
export async function runHook(kind, payload, { root, dryRun = false, timeoutMs = DEFAULT_HOOK_TIMEOUT_MS }) {
|
||||
await loadDotenvOnce();
|
||||
const loaded = await loadPlugins(kind, { root, dryRun });
|
||||
const results = [];
|
||||
for (const { id, hook, ctx } of loaded) {
|
||||
const invoke = kind === 'search'
|
||||
? Promise.resolve().then(() => hook(payload, ctx))
|
||||
: kind === 'export' || kind === 'notify'
|
||||
? Promise.resolve().then(() => hook(payload, ctx))
|
||||
: Promise.resolve().then(() => hook(ctx)); // ingest
|
||||
let timer;
|
||||
const timeout = new Promise((_, rej) => { timer = setTimeout(() => rej(new Error(`timed out after ${timeoutMs}ms`)), timeoutMs); });
|
||||
try {
|
||||
const result = await Promise.race([invoke, timeout]);
|
||||
results.push({ id, ok: true, result });
|
||||
} catch (err) {
|
||||
warnSkip(id, `${kind} hook failed — ${err.message}`);
|
||||
results.push({ id, ok: false, error: err.message });
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge enabled provider-plugins into the core providers Map. THE one hot-path
|
||||
* hook (scan.mjs calls this right after loadProviders). Critical guarantees:
|
||||
*
|
||||
* 1. BYTE-IDENTICAL when opted out: returns IMMEDIATELY if config/plugins.yml
|
||||
* is absent — no discovery, no dotenv, no process.env mutation.
|
||||
* 2. dotenv is loaded LAZILY and only after at least one enabled provider
|
||||
* plugin is found, so a present-but-provider-less plugins.yml still touches
|
||||
* no env.
|
||||
* 3. Core providers ALWAYS win an id collision (a plugin can never shadow a
|
||||
* bundled zero-key provider).
|
||||
* 4. detect-EXEMPT: a merged provider's detect() is forced to null, so it fires
|
||||
* ONLY on an explicit `provider: <id>` portals.yml entry — never via
|
||||
* auto-detection (no surprise paid/keyed network during a plain scan).
|
||||
* 5. A known-but-inactive provider plugin registers a STUB whose fetch throws
|
||||
* an actionable message (disabled / missing key) — so `provider: apify` with
|
||||
* the plugin off yields a helpful error, not a confusing "unknown provider".
|
||||
*
|
||||
* @param {Map<string, any>} providersMap The Map returned by scan.mjs loadProviders.
|
||||
* @param {{ root: string }} opts
|
||||
*/
|
||||
// A detect-exempt provider whose fetch throws an actionable message — used when
|
||||
// a known provider plugin is inactive (disabled / missing key / failed import)
|
||||
// so an explicit `provider: <id>` portals.yml entry stays self-explaining.
|
||||
function inactiveProviderStub(id, reason) {
|
||||
return {
|
||||
id,
|
||||
detect: () => null,
|
||||
fetch: async () => { throw new Error(`plugin "${id}" inactive: ${reason}. Run \`node doctor.mjs\` for setup.`); },
|
||||
};
|
||||
}
|
||||
|
||||
export async function mergeProviderPlugins(providersMap, { root }) {
|
||||
if (!existsSync(pluginsConfigPath(root))) return; // (1) opted out → inert (no work, no env read)
|
||||
|
||||
// Everything past the opt-out gate is wrapped so an UNANTICIPATED throw
|
||||
// (a callee regression) degrades to a ⚠️ and leaves the core providers Map
|
||||
// untouched — fail-open is enforced structurally here, not just emergently.
|
||||
try {
|
||||
const cfg = await loadPluginConfig(root);
|
||||
const providerManifests = discoverPlugins(pluginRoots(root), resolveSuccessorIds(root)).filter(m => m.hooks.includes('provider'));
|
||||
if (providerManifests.length === 0) return;
|
||||
|
||||
// Only the plugins the user actually switched on in plugins.yml matter.
|
||||
const configuredOn = providerManifests.filter(m => cfg?.plugins?.[m.id]?.enabled === true);
|
||||
if (configuredOn.length === 0) return;
|
||||
|
||||
await loadDotenvOnce(); // (2) lazy, only now that an enabled provider plugin exists
|
||||
|
||||
for (const manifest of configuredOn) {
|
||||
if (providersMap.has(manifest.id)) {
|
||||
warnSkip(manifest.id, 'a core provider already owns this id — plugin not merged');
|
||||
continue;
|
||||
}
|
||||
const { enabled, missingEnv } = pluginStatus(manifest, cfg);
|
||||
if (!enabled) {
|
||||
const reason = missingEnv.length ? `missing env ${missingEnv.join(', ')} — add to .env` : 'disabled in config/plugins.yml';
|
||||
providersMap.set(manifest.id, inactiveProviderStub(manifest.id, reason)); // (5)
|
||||
continue;
|
||||
}
|
||||
if (!lockGate(manifest, root).load) {
|
||||
providersMap.set(manifest.id, inactiveProviderStub(manifest.id, 'integrity/consent check failed — see ⚠️ above; run `node plugins.mjs trust ' + manifest.id + '` or `enable ' + manifest.id + '`'));
|
||||
continue;
|
||||
}
|
||||
const hook = await importHook(manifest, 'provider');
|
||||
if (!hook) {
|
||||
// enabled + keyed but the entry failed to import — keep the path self-explaining.
|
||||
providersMap.set(manifest.id, inactiveProviderStub(manifest.id, 'failed to load — see ⚠️ above'));
|
||||
continue;
|
||||
}
|
||||
const ctx = buildCtx(manifest, { settings: pluginSettings(manifest.id, cfg) });
|
||||
providersMap.set(manifest.id, {
|
||||
id: manifest.id,
|
||||
detect: () => null, // (4) keyed providers never auto-detect
|
||||
fetch: (entry) => hook.fetch(entry, ctx), // (3) ctx injection
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
warnSkip('plugins', `provider-plugin merge skipped this run — ${err.message}`);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
// @ts-check
|
||||
// plugins/_lock.mjs — integrity pinning for plugins (the rug-pull / OWASP-MCP04
|
||||
// defense). Pure, side-effect-free import (node:crypto + node:fs only, NO
|
||||
// child_process so it stays loadable under plugins/). plugins.lock is a USER-
|
||||
// layer file (gitignored) recording, per enabled plugin, the sha256 of EVERY
|
||||
// file in its directory + the consented capability surface.
|
||||
//
|
||||
// Honest scope: the lock is tamper-EVIDENCE, not containment. A local attacker
|
||||
// with write access to plugins.local/ also has write access to plugins.lock and
|
||||
// can re-pin. Its real value: catching a bundled-update tamper, a cross-plugin
|
||||
// mutation, and a community plugin whose files changed since you trusted it
|
||||
// WITHOUT a version bump (the postmark-mcp class). Stated plainly in README.
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import { existsSync, readFileSync, writeFileSync, readdirSync, lstatSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
const LOCK_VERSION = 1;
|
||||
|
||||
export function lockPath(root) {
|
||||
return path.join(root, 'plugins.lock');
|
||||
}
|
||||
|
||||
function sha256(buf) {
|
||||
return 'sha256-' + createHash('sha256').update(buf).digest('hex');
|
||||
}
|
||||
|
||||
/**
|
||||
* Hash EVERY regular file in a plugin directory tree, recursively. NOT a curated
|
||||
* subset — the entry can `import('./anything.mjs')`, so a partial hash would let
|
||||
* a rug-pull mutate an un-hashed file. Rejects symlinks (a symlinked file would
|
||||
* pass the hash while pointing elsewhere). Excludes node_modules + .git.
|
||||
*
|
||||
* @param {string} dir absolute plugin directory
|
||||
* @returns {{ files: Record<string,string>, integrity: string }}
|
||||
*/
|
||||
export function hashPluginTree(dir) {
|
||||
const files = {};
|
||||
const walk = (abs, rel) => {
|
||||
let entries;
|
||||
try { entries = readdirSync(abs, { withFileTypes: true }); }
|
||||
catch (err) { throw new Error(`cannot read ${rel || '.'}: ${err.message}`); }
|
||||
for (const e of entries.sort((a, b) => a.name.localeCompare(b.name))) {
|
||||
if (e.name === 'node_modules' || e.name === '.git') continue;
|
||||
const childAbs = path.join(abs, e.name);
|
||||
const childRel = rel ? `${rel}/${e.name}` : e.name;
|
||||
// lstat (not stat) so a symlink is detected, never followed.
|
||||
const st = lstatSync(childAbs);
|
||||
if (st.isSymbolicLink()) throw new Error(`refusing to hash symlink: ${childRel}`);
|
||||
if (st.isDirectory()) walk(childAbs, childRel);
|
||||
else if (st.isFile()) files[childRel] = sha256(readFileSync(childAbs));
|
||||
else throw new Error(`refusing to hash non-regular file: ${childRel}`);
|
||||
}
|
||||
};
|
||||
walk(dir, '');
|
||||
// Aggregate integrity = sha256 over the deterministic sorted "rel:hash" join.
|
||||
const aggregate = Object.keys(files).sort().map(k => `${k}:${files[k]}`).join('\n');
|
||||
return { files, integrity: sha256(Buffer.from(aggregate)) };
|
||||
}
|
||||
|
||||
/** Read plugins.lock (fail-open to an empty lock — like the rest of the engine). */
|
||||
export function readLock(root) {
|
||||
const file = lockPath(root);
|
||||
if (!existsSync(file)) return { lockfileVersion: LOCK_VERSION, plugins: {} };
|
||||
try {
|
||||
const parsed = JSON.parse(readFileSync(file, 'utf8'));
|
||||
if (!parsed || typeof parsed !== 'object' || typeof parsed.plugins !== 'object') return { lockfileVersion: LOCK_VERSION, plugins: {} };
|
||||
return parsed;
|
||||
} catch {
|
||||
return { lockfileVersion: LOCK_VERSION, plugins: {} };
|
||||
}
|
||||
}
|
||||
|
||||
/** Merge a single entry into plugins.lock (never clobber other plugins' entries). */
|
||||
export function writeLockEntry(root, id, entry) {
|
||||
const lock = readLock(root);
|
||||
lock.lockfileVersion = LOCK_VERSION;
|
||||
lock.plugins[id] = entry;
|
||||
writeFileSync(lockPath(root), JSON.stringify(lock, null, 2) + '\n', 'utf8');
|
||||
}
|
||||
|
||||
export function removeLockEntry(root, id) {
|
||||
const lock = readLock(root);
|
||||
if (lock.plugins[id]) { delete lock.plugins[id]; writeFileSync(lockPath(root), JSON.stringify(lock, null, 2) + '\n', 'utf8'); }
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare a plugin's current tree + surface against its recorded lock entry.
|
||||
* `source` is derived by the CALLER from the filesystem (bundled vs local) —
|
||||
* NEVER trusted from the lock (a USER-writable file could spoof it).
|
||||
*
|
||||
* @returns {{ status: 'unpinned'|'match'|'drift-nobump'|'legit-update'|'surface-widened', changedFiles: string[], addedHosts: string[], addedEnv: string[] }}
|
||||
*/
|
||||
export function diffPlugin(manifest, lockEntry) {
|
||||
if (!lockEntry) return { status: 'unpinned', changedFiles: [], addedHosts: [], addedEnv: [] };
|
||||
const tree = hashPluginTree(manifest.dir);
|
||||
const addedHosts = (manifest.allowedHosts || []).filter(h => !(lockEntry.consent?.allowedHosts || []).includes(h));
|
||||
const addedEnv = (manifest.requiredEnv || []).filter(e => !(lockEntry.consent?.requiredEnv || []).includes(e));
|
||||
const surfaceWidened = addedHosts.length > 0 || addedEnv.length > 0 ||
|
||||
(manifest.allowsLocalhost === true && lockEntry.consent?.allowsLocalhost !== true);
|
||||
|
||||
if (tree.integrity === lockEntry.integrity) {
|
||||
return surfaceWidened
|
||||
? { status: 'surface-widened', changedFiles: [], addedHosts, addedEnv }
|
||||
: { status: 'match', changedFiles: [], addedHosts, addedEnv };
|
||||
}
|
||||
// Files changed. Which?
|
||||
const prev = lockEntry.files || {};
|
||||
const changedFiles = [...new Set([...Object.keys(tree.files), ...Object.keys(prev)])]
|
||||
.filter(f => tree.files[f] !== prev[f]);
|
||||
if (surfaceWidened) return { status: 'surface-widened', changedFiles, addedHosts, addedEnv };
|
||||
// A version bump distinguishes an honest update from a stealth mutation.
|
||||
const bumped = compareVersions(manifest.version, lockEntry.version) > 0;
|
||||
return { status: bumped ? 'legit-update' : 'drift-nobump', changedFiles, addedHosts, addedEnv };
|
||||
}
|
||||
|
||||
function compareVersions(a, b) {
|
||||
const pa = String(a || '0').split('.').map(n => parseInt(n, 10) || 0);
|
||||
const pb = String(b || '0').split('.').map(n => parseInt(n, 10) || 0);
|
||||
for (let i = 0; i < 3; i++) { if ((pa[i] || 0) !== (pb[i] || 0)) return (pa[i] || 0) - (pb[i] || 0); }
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** Build the consent surface recorded in a lock entry. */
|
||||
export function consentSurface(manifest) {
|
||||
return {
|
||||
hooks: [...manifest.hooks],
|
||||
requiredEnv: [...manifest.requiredEnv],
|
||||
allowedHosts: [...manifest.allowedHosts],
|
||||
skill: Boolean(manifest.skill),
|
||||
allowsLocalhost: manifest.allowsLocalhost === true,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
// @ts-check
|
||||
// plugins/_net.mjs — SSRF egress validation for the plugin guarded fetch.
|
||||
//
|
||||
// Pure, side-effect-free import (no top-level work) so the engine's
|
||||
// byte-identical-when-opted-out guarantee is preserved. Uses ONLY node builtins
|
||||
// (node:dns, node:net) — NOT undici (Node's global fetch is powered by an
|
||||
// INTERNAL undici that is not importable as the `undici` module in a zero-dep
|
||||
// install, so we never depend on it).
|
||||
//
|
||||
// Honest scope: this resolves a hostname and rejects private/loopback/link-
|
||||
// local/metadata destinations on EVERY redirect hop. It binds an HONEST plugin
|
||||
// that routes through ctx.fetch. It is tamper-evidence + a footgun guard, NOT
|
||||
// containment against malicious code (which can call node:net directly). A
|
||||
// residual DNS-rebinding window exists because global fetch re-resolves at
|
||||
// connect time; documented plainly in plugins/README.md (no overclaim).
|
||||
|
||||
import { isIP } from 'node:net';
|
||||
import { lookup as dnsLookup } from 'node:dns/promises';
|
||||
|
||||
/**
|
||||
* Is an IP address in a range a plugin must never reach (SSRF targets)?
|
||||
* Rejects loopback, RFC1918 private, link-local + cloud metadata (169.254.x,
|
||||
* incl. 169.254.169.254), CGNAT (100.64/10), unspecified, IPv6 ULA/link-local,
|
||||
* and IPv4-mapped-IPv6 of all the above.
|
||||
* @param {string} ip
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function isBlockedIp(ip) {
|
||||
const fam = isIP(ip);
|
||||
if (fam === 0) return true; // not a valid IP → reject (fail-closed)
|
||||
|
||||
if (fam === 4) return isBlockedV4(ip);
|
||||
|
||||
// IPv6
|
||||
const lower = ip.toLowerCase();
|
||||
if (lower === '::1' || lower === '::') return true; // loopback / unspecified
|
||||
if (lower.startsWith('fe80:')) return true; // link-local
|
||||
if (/^f[cd][0-9a-f][0-9a-f]:/.test(lower)) return true; // fc00::/7 ULA
|
||||
// IPv4-mapped (::ffff:a.b.c.d) → validate the embedded v4
|
||||
const mapped = lower.match(/^::ffff:(\d+\.\d+\.\d+\.\d+)$/);
|
||||
if (mapped) return isBlockedV4(mapped[1]);
|
||||
// ::ffff:hex:hex form
|
||||
const mappedHex = lower.match(/^::ffff:([0-9a-f]{1,4}):([0-9a-f]{1,4})$/);
|
||||
if (mappedHex) {
|
||||
const a = parseInt(mappedHex[1], 16), b = parseInt(mappedHex[2], 16);
|
||||
return isBlockedV4([(a >> 8) & 255, a & 255, (b >> 8) & 255, b & 255].join('.'));
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function isBlockedV4(ip) {
|
||||
const o = ip.split('.').map(Number);
|
||||
if (o.length !== 4 || o.some(n => !Number.isInteger(n) || n < 0 || n > 255)) return true;
|
||||
const [a, b] = o;
|
||||
if (a === 0) return true; // 0.0.0.0/8
|
||||
if (a === 10) return true; // 10/8 private
|
||||
if (a === 127) return true; // loopback
|
||||
if (a === 169 && b === 254) return true; // link-local + 169.254.169.254 metadata
|
||||
if (a === 172 && b >= 16 && b <= 31) return true; // 172.16/12 private
|
||||
if (a === 192 && b === 168) return true; // 192.168/16 private
|
||||
if (a === 100 && b >= 64 && b <= 127) return true; // 100.64/10 CGNAT
|
||||
if (a === 192 && b === 0 && o[2] === 0) return true; // 192.0.0/24
|
||||
if (a >= 224) return true; // multicast / reserved
|
||||
return false;
|
||||
}
|
||||
|
||||
const LOOPBACK_HOSTS = new Set(['localhost', '127.0.0.1', '::1', '[::1]']);
|
||||
|
||||
/**
|
||||
* Resolve a hostname and reject if ANY resolved address is blocked. Returns the
|
||||
* validated addresses. Throws on a blocked or unresolvable host.
|
||||
* @param {string} hostname
|
||||
* @param {{ allowsLocalhost?: boolean }} [opts]
|
||||
* @returns {Promise<string[]>}
|
||||
*/
|
||||
export async function resolveAndValidate(hostname, { allowsLocalhost = false } = {}) {
|
||||
// An IP literal host: validate directly (no DNS).
|
||||
if (isIP(hostname)) {
|
||||
if (isBlockedIp(hostname)) {
|
||||
if (allowsLocalhost && isLoopbackLiteral(hostname)) return [hostname];
|
||||
throw new Error(`plugin egress to ${hostname} is blocked (private/loopback/metadata range)`);
|
||||
}
|
||||
return [hostname];
|
||||
}
|
||||
|
||||
if (allowsLocalhost && LOOPBACK_HOSTS.has(hostname.toLowerCase())) {
|
||||
// Local-AI providers (Ollama/LM Studio). Resolve but allow loopback through.
|
||||
return ['127.0.0.1'];
|
||||
}
|
||||
|
||||
let addrs;
|
||||
try {
|
||||
addrs = await dnsLookup(hostname, { all: true });
|
||||
} catch (err) {
|
||||
throw new Error(`plugin egress: cannot resolve ${hostname} — ${err.message}`);
|
||||
}
|
||||
if (!addrs.length) throw new Error(`plugin egress: ${hostname} resolved to no addresses`);
|
||||
for (const { address } of addrs) {
|
||||
if (isBlockedIp(address)) {
|
||||
if (allowsLocalhost && isLoopbackLiteral(address)) continue;
|
||||
throw new Error(`plugin egress: ${hostname} resolves to a blocked address (${address}) — possible SSRF/rebinding`);
|
||||
}
|
||||
}
|
||||
return addrs.map(a => a.address);
|
||||
}
|
||||
|
||||
function isLoopbackLiteral(ip) {
|
||||
if (ip === '::1') return true;
|
||||
if (isIP(ip) === 4) return ip.split('.')[0] === '127';
|
||||
return false;
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
// @ts-check
|
||||
// plugins/_registry.mjs — the curated community-plugin registry (the trust root).
|
||||
// Pure, side-effect-free import (node:fs only). plugins-registry.json is a
|
||||
// SYSTEM-layer file: an entry exists only because a maintainer reviewed + merged
|
||||
// it at an exact pinned commit. Users only ever install the pinnedSha the
|
||||
// SHIPPED registry names — an author cannot reach users without a merged PR.
|
||||
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
|
||||
export function registryPath(root) {
|
||||
return path.join(root, 'plugins-registry.json');
|
||||
}
|
||||
|
||||
/** Load the curated registry. Fail-open to an empty registry. */
|
||||
export function loadRegistry(root) {
|
||||
const f = registryPath(root);
|
||||
if (!existsSync(f)) return { registryVersion: 1, plugins: [] };
|
||||
try {
|
||||
const p = JSON.parse(readFileSync(f, 'utf8'));
|
||||
return p && Array.isArray(p.plugins) ? p : { registryVersion: 1, plugins: [] };
|
||||
} catch {
|
||||
return { registryVersion: 1, plugins: [] };
|
||||
}
|
||||
}
|
||||
|
||||
/** Find a registry entry by bare id, full name, or `career-ops-plugin-<id>`. */
|
||||
export function findInRegistry(root, nameOrId) {
|
||||
const reg = loadRegistry(root);
|
||||
return reg.plugins.find(p =>
|
||||
p.id === nameOrId || p.name === nameOrId || p.name === `career-ops-plugin-${nameOrId}`) || null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a discovered plugin's trust state. Source is derived from the
|
||||
* FILESYSTEM (bundled = under plugins/) and the live registry — NEVER from the
|
||||
* USER-writable plugins.lock.
|
||||
* - bundled : shipped in plugins/, reviewed in-tree
|
||||
* - approved : in plugins.local/ AND its installed sha matches a registry entry
|
||||
* - off-registry: installed from a community repo, but a DIFFERENT commit than the approved one
|
||||
* - unverified : installed locally, not in the registry (user chose to trust the author)
|
||||
* @returns {'bundled'|'approved'|'off-registry'|'unverified'}
|
||||
*/
|
||||
export function classifySource(manifest, root, lockEntry) {
|
||||
if (manifest.dir.startsWith(path.join(root, 'plugins') + path.sep)) return 'bundled';
|
||||
const reg = findInRegistry(root, manifest.id);
|
||||
if (!reg) return 'unverified';
|
||||
if (lockEntry && lockEntry.sha && reg.sha && lockEntry.sha !== reg.sha) return 'off-registry';
|
||||
return 'approved';
|
||||
}
|
||||
|
||||
/** Human-facing badge for a trust source. */
|
||||
export function sourceBadge(source) {
|
||||
return {
|
||||
bundled: '📦 bundled',
|
||||
approved: '✓ approved',
|
||||
'off-registry': '⚠️ off-registry',
|
||||
unverified: '❓ community-unverified',
|
||||
}[source] || source;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate one registry entry's shape (used by the registry CI + a test-all
|
||||
* assertion). Returns an array of problems (empty = valid). `idRe` + `hookKinds`
|
||||
* + `reservedEnv` are passed in from the engine to keep this module dep-free.
|
||||
*/
|
||||
export function validateRegistryEntry(e, { idRe, hookKinds, reservedEnv }) {
|
||||
const errs = [];
|
||||
if (typeof e.name !== 'string' || !e.name.startsWith('career-ops-plugin-')) errs.push('name must start with "career-ops-plugin-"');
|
||||
if (typeof e.id !== 'string' || !idRe.test(e.id)) errs.push('invalid id');
|
||||
if (e.name && e.id && e.name !== `career-ops-plugin-${e.id}`) errs.push('name must equal career-ops-plugin-<id>');
|
||||
if (!/^https:\/\/github\.com\/[^/]+\/[^/]+$/.test(e.repo || '')) errs.push('repo must be a https://github.com/<owner>/<repo> URL');
|
||||
if (!/^[0-9a-f]{40}$/.test(e.sha || '')) errs.push('sha must be a 40-hex commit');
|
||||
if (!Array.isArray(e.hooks) || e.hooks.length === 0 || e.hooks.some(h => !hookKinds.includes(h))) errs.push('hooks must be a non-empty subset of the hook kinds');
|
||||
if (!Array.isArray(e.requiredEnv)) errs.push('requiredEnv must be an array');
|
||||
else if (e.requiredEnv.some(n => reservedEnv.has(n) || /^AWS_/.test(n))) errs.push('requiredEnv declares a reserved/core-owned var');
|
||||
if (!Array.isArray(e.allowedHosts)) errs.push('allowedHosts must be an array');
|
||||
else if (Array.isArray(e.requiredEnv) && e.requiredEnv.length > 0 && e.allowedHosts.length === 0) errs.push('a keyed plugin must declare allowedHosts');
|
||||
if (typeof e.license !== 'string' || !e.license) errs.push('license is required');
|
||||
if (typeof e.version !== 'string') errs.push('version is required');
|
||||
// Optional successor marker: a community plugin can declare itself the
|
||||
// maintained successor of the bundled plugin with the SAME id. When present
|
||||
// it must be the literal boolean true; the engine only grants it precedence
|
||||
// once the user installs it at this exact pinned sha (resolveSuccessorIds).
|
||||
if (e.supersedesBundled !== undefined && e.supersedesBundled !== true) errs.push('supersedesBundled, if present, must be the boolean true');
|
||||
return errs;
|
||||
}
|
||||
|
||||
/**
|
||||
* The registry entry that is the maintained successor of a bundled plugin id
|
||||
* (declares `supersedesBundled: true` and shares the id), or null. Used only to
|
||||
* SURFACE the relationship in `plugins.mjs list`/`available` — precedence itself
|
||||
* is decided by the engine's resolveSuccessorIds() (which also checks install + sha).
|
||||
*/
|
||||
export function successorFor(root, bundledId) {
|
||||
return loadRegistry(root).plugins.find(p => p.supersedesBundled === true && p.id === bundledId) || null;
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) the career-ops-plugin-{{NAME}} authors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,38 @@
|
||||
# career-ops-plugin-{{NAME}}
|
||||
|
||||
A community plugin for [career-ops](https://github.com/santifer/career-ops).
|
||||
|
||||
## What it does
|
||||
|
||||
TODO: one paragraph.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Once it's in the career-ops registry:
|
||||
node plugins.mjs add {{NAME}}
|
||||
|
||||
# Before listing (install directly from your repo at a pinned commit):
|
||||
node plugins.mjs add <your-github-user>/career-ops-plugin-{{NAME}} --sha <40-hex-commit>
|
||||
```
|
||||
|
||||
Then enable + consent:
|
||||
|
||||
```bash
|
||||
node plugins.mjs enable {{NAME}} # shows the capability card
|
||||
node plugins.mjs enable {{NAME}} --confirm # grants it
|
||||
```
|
||||
|
||||
## Configure
|
||||
|
||||
- Secrets go in your `.env` (the names are in `manifest.json` → `requiredEnv`).
|
||||
- Non-secret options go in `config/plugins.yml` under `plugins.{{NAME}}`.
|
||||
|
||||
## Get it listed as approved
|
||||
|
||||
Open a registry PR against career-ops (see
|
||||
[docs/PLUGINS.md](https://github.com/santifer/career-ops/blob/main/docs/PLUGINS.md)).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,22 @@
|
||||
// @ts-check
|
||||
// {{NAME}} — a career-ops plugin.
|
||||
// Guide: https://github.com/santifer/career-ops/blob/main/docs/PLUGINS.md
|
||||
//
|
||||
// Rules the engine enforces for you:
|
||||
// - Egress ONLY through ctx.fetch / ctx.fetchJson / ctx.fetchText (your manifest
|
||||
// `allowedHosts` is applied + SSRF-guarded). Do NOT import node:http/net or
|
||||
// call global fetch — community plugins are rejected for that.
|
||||
// - Producers (provider/ingest/search) RETURN Job[] = { title, url, company, location };
|
||||
// the engine writes them to the pipeline. Consumers (export/notify) push to
|
||||
// the user's own store. There is no auto-submit hook.
|
||||
// - Keys come from ctx.env (declare them in manifest.requiredEnv); non-secret
|
||||
// settings come from ctx.settings (the user's config/plugins.yml block).
|
||||
|
||||
export default {
|
||||
// Replace/add hooks to match manifest.hooks. Example ingest:
|
||||
async ingest(ctx) {
|
||||
// const data = await ctx.fetchJson('https://api.example.com/jobs');
|
||||
// return data.results.map(j => ({ title: j.title, url: j.url, company: j.company, location: j.location || '' }));
|
||||
return [];
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"id": "{{NAME}}",
|
||||
"name": "{{NAME}}",
|
||||
"version": "0.1.0",
|
||||
"apiVersion": 1,
|
||||
"description": "TODO: one mission-framed line describing what this plugin does.",
|
||||
"hooks": ["ingest"],
|
||||
"requiredEnv": [],
|
||||
"allowedHosts": [],
|
||||
"skill": "skill.md",
|
||||
"humanInTheLoop": true,
|
||||
"homepage": "https://github.com/your-user/career-ops-plugin-{{NAME}}"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: career-ops-plugin-{{NAME}}
|
||||
description: How to use the {{NAME}} plugin and the data it produces.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# {{NAME}}
|
||||
|
||||
> This file teaches an AI agent how to drive THIS plugin. Keep it scoped to the
|
||||
> plugin's own domain — it must not instruct the agent to edit core files, change
|
||||
> scoring, or act outside the plugin's declared hooks.
|
||||
|
||||
## How to run it
|
||||
|
||||
- `node plugins.mjs run {{NAME}}` — runs the plugin's hook.
|
||||
|
||||
## What it produces
|
||||
|
||||
TODO: describe the data structure. For a producer hook, the `Job[]` fields you
|
||||
emit (title, url, company, location). For an export hook, what you push and where.
|
||||
|
||||
## Settings
|
||||
|
||||
TODO: any non-secret options the user sets in `config/plugins.yml` under
|
||||
`plugins.{{NAME}}` (these arrive as `ctx.settings`).
|
||||
@@ -0,0 +1,21 @@
|
||||
// Zero-network smoke test: the entry imports cleanly and exposes only valid
|
||||
// hooks that match the manifest. Run by `plugins.mjs add` + the registry CI.
|
||||
import assert from 'node:assert';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import path from 'node:path';
|
||||
|
||||
const here = path.dirname(fileURLToPath(import.meta.url));
|
||||
const KINDS = ['provider', 'ingest', 'search', 'notify', 'export'];
|
||||
|
||||
const manifest = JSON.parse(readFileSync(path.join(here, '..', 'manifest.json'), 'utf8'));
|
||||
const mod = await import(path.join(here, '..', manifest.entry || 'index.mjs'));
|
||||
const hooks = mod.default;
|
||||
|
||||
assert(hooks && typeof hooks === 'object', 'default export must be an object of hooks');
|
||||
const keys = Object.keys(hooks);
|
||||
assert(keys.length > 0, 'declare at least one hook');
|
||||
for (const k of keys) assert(KINDS.includes(k), `unknown hook "${k}"`);
|
||||
for (const h of manifest.hooks) assert(keys.includes(h), `manifest declares hook "${h}" but index.mjs does not export it`);
|
||||
|
||||
console.log('✓ smoke ok:', keys.join(', '));
|
||||
@@ -0,0 +1,84 @@
|
||||
// Type catalog for the career-ops plugin contract.
|
||||
//
|
||||
// Documentation-only — pure JSDoc @typedef annotations, no build step (the
|
||||
// project is plain ESM JavaScript). Plugin authors reference these via
|
||||
// `/** @typedef {import('../_types.js').PluginManifest} PluginManifest */` at
|
||||
// the top of a `// @ts-check`-enabled file to get IDE hints. The runtime
|
||||
// contract is enforced by _engine.mjs (validateManifest), not by these
|
||||
// annotations — exactly the providers/_types.js + scan.mjs split.
|
||||
//
|
||||
// Files prefixed with _ are shared helpers and are NEVER discovered as plugins.
|
||||
//
|
||||
// A plugin is a directory under plugins/ (bundled, system layer) or
|
||||
// plugins.local/ (user-installed, gitignored) containing a manifest.json and an
|
||||
// entry module (default index.mjs). The manifest is JSON — parsed, not executed
|
||||
// — so the loader and `doctor` can validate and list a plugin WITHOUT importing
|
||||
// its code. The entry module default-exports an object keyed by hook type.
|
||||
//
|
||||
// The unit of currency for the producer hooks is the SAME `Job` the scanner
|
||||
// uses — see providers/_types.js. Producers RETURN Job[]; the engine (never the
|
||||
// plugin) writes them through scan.mjs's canonical writers, so a plugin can
|
||||
// never break the web-facing data formats.
|
||||
|
||||
/**
|
||||
* The plugin manifest — `plugins/<id>/manifest.json`. Named manifest.json (NOT
|
||||
* plugin.json) to avoid colliding with the repo's existing
|
||||
* `.claude-plugin/plugin.json` (the unrelated Claude Code plugin marketplace).
|
||||
*
|
||||
* @typedef {object} PluginManifest
|
||||
* @property {string} id REQUIRED. ^[a-z0-9][a-z0-9-]*$. MUST equal the directory name (dedup key + anti-spoof).
|
||||
* @property {number} apiVersion REQUIRED. Must be 1. The engine refuses unknown majors so ctx/hook-signature changes can be versioned once external plugins exist.
|
||||
* @property {string} description REQUIRED. One line, mission-framed (no revenue/pricing/moat/hosted wording — it ships in a public repo).
|
||||
* @property {string[]} hooks REQUIRED, non-empty. Subset of {'provider','ingest','search','notify','export'}. Anything else (e.g. 'apply'/'submit') is REJECTED — there is deliberately no auto-submit hook.
|
||||
* @property {string[]} requiredEnv REQUIRED (may be []). Env VARIABLE NAMES only, never values. The user supplies values in their own gitignored .env. A reserved-name denylist (core secrets, PATH, AWS_*, …) is rejected.
|
||||
* @property {string} [name] Optional human label for `doctor`/CLI listings.
|
||||
* @property {string} [version] Optional semver of the plugin itself.
|
||||
* @property {string} [entry] Optional. Default 'index.mjs'. Must be a relative *.mjs path that stays inside the plugin directory (traversal-guarded).
|
||||
* @property {string[]} [optionalEnv] Optional extra env var names (same denylist applies).
|
||||
* @property {string[]} [allowedHosts] Optional egress allowlist. REQUIRED (loader-enforced) when requiredEnv is non-empty. ADVISORY: it catches an honest plugin's own SSRF/redirect bugs when it routes through ctx — it is NOT a containment boundary against malicious code (see the trust note in README.md).
|
||||
* @property {boolean} humanInTheLoop REQUIRED true. The loader hard-rejects false. A plugin may read/ingest/search/notify/export — never auto-submit a job application.
|
||||
* @property {string} [homepage] Optional URL.
|
||||
*/
|
||||
|
||||
/**
|
||||
* The capability object the engine builds per plugin and passes to every hook.
|
||||
* Least-privilege by construction, but a CONVENIENCE not a security sandbox —
|
||||
* plain ESM cannot isolate a module's ambient imports without a build step.
|
||||
* Containment is code review (bundled plugins, same gate as providers/) plus
|
||||
* the user's own trust for plugins.local/. See README.md "Trust model".
|
||||
*
|
||||
* @typedef {object} PluginContext
|
||||
* @property {'http'} transport
|
||||
* @property {(url: string, opts?: object) => Promise<Response>} fetch The guarded primitive (HTTPS-only, allowedHosts-pinned, redirect:'manual' with every hop re-validated + cross-host credential strip). Bundled plugins route HTTP through this so the egress guard actually runs. Throws on non-2xx.
|
||||
* @property {(url: string, opts?: object) => Promise<string>} fetchText Convenience over ctx.fetch → .text().
|
||||
* @property {(url: string, opts?: object) => Promise<unknown>} fetchJson Same guard as fetchText.
|
||||
* @property {Readonly<Object<string,string>>} env Frozen, scoped to this plugin's declared requiredEnv ∪ optionalEnv. A convenience accessor — NOT an isolation boundary (process.env stays globally reachable from any module).
|
||||
* @property {Readonly<Object<string,unknown>>} settings Frozen non-secret settings block for this plugin from config/plugins.yml (e.g. { label, days_back, actor }). Secrets never live here — they go in .env via requiredEnv.
|
||||
* @property {(...args: unknown[]) => void} log Console logger that redacts declared env values from output (accidental-leak hygiene, not an exfiltration control).
|
||||
* @property {boolean} dryRun True when invoked with --dry-run; side-effecting hooks must honor it.
|
||||
*/
|
||||
|
||||
/**
|
||||
* The default export of a plugin's entry module — an object with one function
|
||||
* per declared hook. Only the hooks named in the manifest are loaded.
|
||||
*
|
||||
* - provider: BYTE-IDENTICAL to providers/_types.js Provider, plus ctx.env. A
|
||||
* keyed/auth-gated job source. The engine forces detect() to null so a keyed
|
||||
* provider only fires on an explicit `provider: <id>` portals.yml entry —
|
||||
* never via auto-detection (no surprise paid/keyed network during a scan).
|
||||
* - ingest: pull postings from a service (email, a Notion board) → Job[]. The
|
||||
* engine appends them to data/pipeline.md canonically.
|
||||
* - search: Job[] for a query string → engine writes canonically to pipeline.
|
||||
* - export: receives a frozen read-only snapshot of the user's tracker and
|
||||
* pushes it to the user's OWN external store; returns {pushed}. No file handle.
|
||||
* - notify: outbound, ephemeral notification of a payload.
|
||||
*
|
||||
* @typedef {object} PluginHooks
|
||||
* @property {{ id: string, detect?: (entry: object) => ({url: string}|null), fetch: (entry: object, ctx: PluginContext) => Promise<object[]> }} [provider]
|
||||
* @property {(ctx: PluginContext) => Promise<object[]>} [ingest]
|
||||
* @property {(query: string, ctx: PluginContext) => Promise<object[]>} [search]
|
||||
* @property {(snapshot: Readonly<object>, ctx: PluginContext) => Promise<{pushed: number}|void>} [export]
|
||||
* @property {(payload: Readonly<object>, ctx: PluginContext) => Promise<void>} [notify]
|
||||
*/
|
||||
|
||||
export {};
|
||||
@@ -0,0 +1,202 @@
|
||||
// Apify transport helper — runs pre-built actors and returns dataset items.
|
||||
// Used by the Apify provider plugin (plugins/apify/index.mjs).
|
||||
//
|
||||
// Ported from the `providers/_apify.mjs` contributed by @ageem23 in #693 (with
|
||||
// thanks). The only change for the plugin layer: runActor() takes the token as
|
||||
// an argument (supplied from the plugin's scoped ctx.env) instead of reading
|
||||
// process.env directly. Files prefixed with _ are never discovered as plugins.
|
||||
//
|
||||
// Uses the async pattern (start → poll → fetch dataset) rather than the
|
||||
// long-polling /run-sync-get-dataset-items endpoint, which holds one HTTP
|
||||
// connection open for the full actor run and gets cut by some networks /
|
||||
// Windows SChannel before Apify can flush the response.
|
||||
|
||||
// Egress note: this helper keeps its own fetch (its retry / shared-deadline /
|
||||
// abort logic is the whole point and doesn't map cleanly onto ctx.fetch). It
|
||||
// self-constrains harder than allowedHosts would — every request is built from
|
||||
// this single hardcoded base + a strictly-validated actorId (normalizeActorId),
|
||||
// so it can only ever reach api.apify.com. The manifest's allowedHosts mirrors
|
||||
// that for doctor/review visibility.
|
||||
const APIFY_API_BASE = 'https://api.apify.com/v2';
|
||||
const DEFAULT_RUN_TIMEOUT_MS = 180_000;
|
||||
const POLL_INTERVAL_MS = 3_000;
|
||||
const PER_REQUEST_TIMEOUT_MS = 15_000;
|
||||
const CONNECT_RETRY_ATTEMPTS = 3;
|
||||
const TERMINAL_STATUSES = new Set(['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT']);
|
||||
|
||||
export function hasToken(token = process.env.APIFY_TOKEN) {
|
||||
return Boolean(token);
|
||||
}
|
||||
|
||||
// Apify accepts both "user/actor" and "user~actor" in URLs; normalize to `~`.
|
||||
// Validate strictly so a malformed config can't escape the intended
|
||||
// /acts/<actor>/runs path with extra `/`, `..`, `?`, or `#` characters and
|
||||
// send our bearer token to an unintended endpoint on api.apify.com.
|
||||
const ACTOR_ID_RE = /^[A-Za-z0-9][A-Za-z0-9_.-]*[~/][A-Za-z0-9][A-Za-z0-9_.-]*$/;
|
||||
|
||||
export function normalizeActorId(actorId) {
|
||||
if (typeof actorId !== 'string' || !ACTOR_ID_RE.test(actorId)) {
|
||||
throw new Error(
|
||||
`apify: invalid actorId ${JSON.stringify(actorId)}. ` +
|
||||
`Expected "owner/actor" or "owner~actor" with letters, digits, "_", ".", or "-" only.`
|
||||
);
|
||||
}
|
||||
const [owner, name] = actorId.split(/[~/]/, 2);
|
||||
return `${encodeURIComponent(owner)}~${encodeURIComponent(name)}`;
|
||||
}
|
||||
|
||||
// Apify supports auth via ?token= or Authorization: Bearer. The query-string
|
||||
// form leaks the token into HTTP access logs and any error/log line that
|
||||
// includes the URL, so always use the header.
|
||||
function authHeaders(token) {
|
||||
return { authorization: `Bearer ${token}` };
|
||||
}
|
||||
|
||||
function sleep(ms) {
|
||||
return new Promise(r => setTimeout(r, ms));
|
||||
}
|
||||
|
||||
async function fetchJsonOnce(url, init = {}, timeoutMs = PER_REQUEST_TIMEOUT_MS) {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
||||
try {
|
||||
const res = await fetch(url, { ...init, signal: controller.signal });
|
||||
if (!res.ok) {
|
||||
const text = await res.text().catch(() => '');
|
||||
const err = new Error(`HTTP ${res.status}: ${text.slice(0, 300)}`);
|
||||
err.status = res.status;
|
||||
throw err;
|
||||
}
|
||||
return await res.json();
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
// Cold connects to api.apify.com occasionally exceed Undici's 10s internal
|
||||
// connect timeout on this host — retry with backoff before giving up. When
|
||||
// `deadline` is provided, each attempt's per-request timeout and the
|
||||
// inter-attempt backoff are both capped against the remaining budget.
|
||||
async function fetchJson(
|
||||
url,
|
||||
init = {},
|
||||
timeoutMs = PER_REQUEST_TIMEOUT_MS,
|
||||
attempts = CONNECT_RETRY_ATTEMPTS,
|
||||
deadline = null,
|
||||
) {
|
||||
let lastErr;
|
||||
for (let i = 0; i < attempts; i++) {
|
||||
let thisTimeout = timeoutMs;
|
||||
if (deadline != null) {
|
||||
const remainingMs = deadline - Date.now();
|
||||
if (remainingMs <= 0) throw lastErr || new Error('Apify request budget exhausted');
|
||||
thisTimeout = Math.min(timeoutMs, remainingMs);
|
||||
}
|
||||
try {
|
||||
return await fetchJsonOnce(url, init, thisTimeout);
|
||||
} catch (err) {
|
||||
lastErr = err;
|
||||
if (err.status >= 400 && err.status < 500) throw err;
|
||||
if (i < attempts - 1) {
|
||||
const backoff = 500 * (i + 1);
|
||||
const sleepMs = deadline != null ? Math.min(backoff, deadline - Date.now()) : backoff;
|
||||
if (sleepMs > 0) await sleep(sleepMs);
|
||||
else if (deadline != null) throw lastErr;
|
||||
}
|
||||
}
|
||||
}
|
||||
throw lastErr;
|
||||
}
|
||||
|
||||
async function startRun(actorId, input, token, deadline = null) {
|
||||
const url = `${APIFY_API_BASE}/acts/${normalizeActorId(actorId)}/runs`;
|
||||
const body = await fetchJson(
|
||||
url,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json', ...authHeaders(token) },
|
||||
body: JSON.stringify(input || {}),
|
||||
},
|
||||
PER_REQUEST_TIMEOUT_MS,
|
||||
CONNECT_RETRY_ATTEMPTS,
|
||||
deadline,
|
||||
);
|
||||
const runId = body?.data?.id;
|
||||
if (!runId) {
|
||||
throw new Error(`Apify did not return a run id: ${JSON.stringify(body).slice(0, 200)}`);
|
||||
}
|
||||
return runId;
|
||||
}
|
||||
|
||||
// Best-effort — if we give up on a run, stop the actor so credits aren't wasted.
|
||||
async function abortRun(runId, token) {
|
||||
const url = `${APIFY_API_BASE}/actor-runs/${runId}/abort`;
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), 5_000);
|
||||
try {
|
||||
await fetch(url, { method: 'POST', headers: authHeaders(token), signal: controller.signal });
|
||||
} catch {} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
async function waitForRun(runId, token, deadline, timeoutMs) {
|
||||
const url = `${APIFY_API_BASE}/actor-runs/${runId}`;
|
||||
let lastError;
|
||||
while (Date.now() < deadline) {
|
||||
const remainingMs = deadline - Date.now();
|
||||
if (remainingMs <= 0) break;
|
||||
try {
|
||||
const body = await fetchJsonOnce(
|
||||
url,
|
||||
{ headers: authHeaders(token) },
|
||||
Math.min(PER_REQUEST_TIMEOUT_MS, remainingMs),
|
||||
);
|
||||
const run = body?.data;
|
||||
if (run && TERMINAL_STATUSES.has(run.status)) return run;
|
||||
lastError = undefined;
|
||||
} catch (err) {
|
||||
// 4xx (401/403 auth revoked, 404 run not found) won't succeed on retry.
|
||||
if (err?.status >= 400 && err.status < 500) throw err;
|
||||
lastError = err;
|
||||
}
|
||||
const sleepMs = Math.min(POLL_INTERVAL_MS, deadline - Date.now());
|
||||
if (sleepMs > 0) await sleep(sleepMs);
|
||||
}
|
||||
// Fire-and-forget cleanup; don't add abortRun's 5s to our wall-clock budget.
|
||||
void abortRun(runId, token).catch(() => {});
|
||||
const suffix = lastError ? ` (last error: ${lastError.message})` : '';
|
||||
throw new Error(`Apify run ${runId} did not finish within ${Math.round(timeoutMs / 1000)}s${suffix}`);
|
||||
}
|
||||
|
||||
async function fetchDatasetItems(runId, token, deadline = null) {
|
||||
const url = `${APIFY_API_BASE}/actor-runs/${runId}/dataset/items`;
|
||||
const items = await fetchJson(
|
||||
url,
|
||||
{ headers: authHeaders(token) },
|
||||
PER_REQUEST_TIMEOUT_MS * 2,
|
||||
CONNECT_RETRY_ATTEMPTS,
|
||||
deadline,
|
||||
);
|
||||
if (!Array.isArray(items)) {
|
||||
throw new Error(`Apify run ${runId} returned non-array dataset payload`);
|
||||
}
|
||||
return items;
|
||||
}
|
||||
|
||||
export async function runActor(actorId, input, { timeoutMs = DEFAULT_RUN_TIMEOUT_MS, token = process.env.APIFY_TOKEN } = {}) {
|
||||
if (!token) throw new Error('APIFY_TOKEN not set');
|
||||
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) {
|
||||
throw new Error(`apify: invalid timeoutMs ${JSON.stringify(timeoutMs)} (must be a positive finite number of milliseconds)`);
|
||||
}
|
||||
// Single deadline shared across startRun → waitForRun → fetchDatasetItems so
|
||||
// the caller's timeoutMs is the end-to-end ceiling, not just the wait loop.
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
const runId = await startRun(actorId, input, token, deadline);
|
||||
const run = await waitForRun(runId, token, deadline, timeoutMs);
|
||||
if (run.status !== 'SUCCEEDED') {
|
||||
const reason = run.statusMessage ? `: ${run.statusMessage}` : '';
|
||||
throw new Error(`Apify actor ${actorId} finished with status ${run.status}${reason}`);
|
||||
}
|
||||
return await fetchDatasetItems(runId, token, deadline);
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
// @ts-check
|
||||
// ── Reference seed ── This bundled plugin is a stable, reviewed example. To
|
||||
// extend it, publish career-ops-plugin-<id> with "supersedesBundled": true and
|
||||
// your version takes precedence once installed (see docs/PLUGINS.md). Bundled
|
||||
// seeds take only security/compat fixes — feature work happens in the successor repo.
|
||||
//
|
||||
// Apify provider plugin — runs any Apify actor and maps its dataset items to
|
||||
// the {title, url, company, location} Job shape the scanner expects. All
|
||||
// variation (which actor, what input, how to read fields) lives in portals.yml.
|
||||
//
|
||||
// Ported from the generic Apify provider contributed by @ageem23 in #693 (with
|
||||
// thanks); it also homes the LinkedIn-via-Apify use case from #791/#1202. As a
|
||||
// KEYED provider it lives here in plugins/ (not the zero-key providers/ dir):
|
||||
// enable it in config/plugins.yml and put APIFY_TOKEN in .env. It fires ONLY on
|
||||
// a portals.yml entry that sets `provider: apify` — never via auto-detection.
|
||||
//
|
||||
// tracked_companies:
|
||||
// - name: "Indeed — VP Engineering (Chicago)"
|
||||
// provider: apify
|
||||
// actor: misceres/indeed-scraper
|
||||
// input: { position: "VP of Engineering", location: "Chicago, IL", maxItems: 25 }
|
||||
// field_map:
|
||||
// title: [positionName, title] # array = first non-empty wins
|
||||
// url: url
|
||||
// company: [company, companyName]
|
||||
// location: [location, formattedLocation]
|
||||
|
||||
import { mkdirSync, writeFileSync, existsSync } from 'fs';
|
||||
import { createHash } from 'crypto';
|
||||
import { join } from 'path';
|
||||
import { hasToken, runActor } from './_apify.mjs';
|
||||
|
||||
const JDS_DIR = 'jds';
|
||||
const MIN_JD_BODY_CHARS = 50;
|
||||
|
||||
function getPath(obj, p) {
|
||||
return p.split('.').reduce((o, k) => (o == null ? undefined : o[k]), obj);
|
||||
}
|
||||
|
||||
// A valid field_map entry is a single key ('positionName') or an ordered list
|
||||
// of fallback keys (['positionName', 'title']). Reject any other shape at
|
||||
// config-load time with a clear error instead of crashing mid-scan.
|
||||
export function isFieldSpec(spec) {
|
||||
if (typeof spec === 'string') return true;
|
||||
if (Array.isArray(spec) && spec.length > 0 && spec.every(s => typeof s === 'string')) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
function pickField(item, spec) {
|
||||
const keys = Array.isArray(spec) ? spec : [spec];
|
||||
for (const k of keys) {
|
||||
const v = getPath(item, k);
|
||||
if (v != null && v !== '') return v;
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
const ALLOWED_DEFAULT_KEYS = new Set(['title', 'url', 'company', 'location']);
|
||||
|
||||
// Actors return URLs from arbitrary external sites — treat them as untrusted.
|
||||
// Reject anything that isn't https so javascript:/data:/file:/http: URLs can't
|
||||
// end up clickable in pipeline.md or in the JD-cache filename hash.
|
||||
export function isHttpsUrl(value) {
|
||||
try {
|
||||
return new URL(String(value)).protocol === 'https:';
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function slugify(text) {
|
||||
const slug = String(text || '')
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, '-')
|
||||
.replace(/^-+|-+$/g, '')
|
||||
.slice(0, 80);
|
||||
if (slug) return slug;
|
||||
const hash = createHash('sha1').update(String(text || '')).digest('hex').slice(0, 10);
|
||||
return `jd-${hash}`;
|
||||
}
|
||||
|
||||
function yamlEscape(str) {
|
||||
const s = String(str ?? '').replace(/\n/g, ' ').trim();
|
||||
return `"${s.replace(/\\/g, '\\\\').replace(/"/g, '\\"')}"`;
|
||||
}
|
||||
|
||||
// Lightweight HTML → text for actor description fields (enough for downstream
|
||||
// snippet extraction and full evaluation, not a full parser).
|
||||
export function htmlToText(s) {
|
||||
const raw = String(s || '');
|
||||
if (!raw || !/[<&]/.test(raw)) return raw.trim();
|
||||
let cleaned = raw
|
||||
.replace(/<script\b[\s\S]*?<\/script\b[^>]*>/gi, ' ')
|
||||
.replace(/<style\b[\s\S]*?<\/style\b[^>]*>/gi, ' ')
|
||||
.replace(/<br\s*\/?>/gi, '\n')
|
||||
.replace(/<\/p>/gi, '\n\n')
|
||||
.replace(/<li[^>]*>/gi, '- ')
|
||||
.replace(/<\/li>/gi, '\n');
|
||||
let prev;
|
||||
do {
|
||||
prev = cleaned;
|
||||
cleaned = cleaned.replace(/<[^>]+>/g, '');
|
||||
} while (cleaned !== prev);
|
||||
return cleaned
|
||||
// Decode & LAST so `&#60;` round-trips to `<` not `<`.
|
||||
.replace(/ /g, ' ')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>')
|
||||
.replace(/"/g, '"')
|
||||
.replace(/'/g, "'")
|
||||
.replace(/&#(\d+);/g, (_, n) => String.fromCharCode(parseInt(n, 10)))
|
||||
.replace(/&/g, '&')
|
||||
.replace(/[ \t]+/g, ' ')
|
||||
.replace(/\n{3,}/g, '\n\n')
|
||||
.trim();
|
||||
}
|
||||
|
||||
// Write jds/{slug}-{hash}.md and return its relative path. The URL-derived hash
|
||||
// keeps two distinct postings sharing a company+title from colliding. Atomic
|
||||
// (flag:'wx') against the 10-worker TOCTOU race; any FS failure returns null so
|
||||
// the caller falls back to the remote URL.
|
||||
function saveJd(normalized, descriptionBody, sourceLabel) {
|
||||
let relPath = null;
|
||||
try {
|
||||
mkdirSync(JDS_DIR, { recursive: true });
|
||||
const baseSlug = slugify(`${normalized.company}-${normalized.title}`);
|
||||
const urlHash = createHash('sha1')
|
||||
.update(String(normalized.url || `${normalized.company}-${normalized.title}`))
|
||||
.digest('hex')
|
||||
.slice(0, 10);
|
||||
const filename = `${baseSlug}-${urlHash}.md`;
|
||||
const filepath = join(JDS_DIR, filename);
|
||||
relPath = `${JDS_DIR}/${filename}`;
|
||||
if (existsSync(filepath)) return relPath;
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const content = `---
|
||||
title: ${yamlEscape(normalized.title)}
|
||||
company: ${yamlEscape(normalized.company)}
|
||||
url: ${yamlEscape(normalized.url)}
|
||||
location: ${yamlEscape(normalized.location)}
|
||||
scraped: "${today}"
|
||||
source: ${sourceLabel}
|
||||
---
|
||||
|
||||
# ${normalized.title} — ${normalized.company}
|
||||
|
||||
${descriptionBody}
|
||||
`;
|
||||
writeFileSync(filepath, content, { encoding: 'utf-8', flag: 'wx' });
|
||||
return relPath;
|
||||
} catch (err) {
|
||||
if (err?.code === 'EEXIST' && relPath) return relPath;
|
||||
console.warn(`apify: JD cache write failed for ${normalized.title} (${err.code || err.name}: ${err.message}); falling back to remote URL`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function normalizeItem(item, fieldMap, defaults) {
|
||||
const out = {
|
||||
title: String(pickField(item, fieldMap.title) || ''),
|
||||
url: String(pickField(item, fieldMap.url) || ''),
|
||||
company: fieldMap.company ? String(pickField(item, fieldMap.company) || '') : '',
|
||||
location: fieldMap.location ? String(pickField(item, fieldMap.location) || '') : '',
|
||||
};
|
||||
for (const [k, v] of Object.entries(defaults || {})) {
|
||||
if (!ALLOWED_DEFAULT_KEYS.has(k)) continue;
|
||||
if (!out[k]) out[k] = String(v);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The keyed provider hook. Reads APIFY_TOKEN from the plugin's scoped ctx.env. */
|
||||
export default {
|
||||
provider: {
|
||||
id: 'apify',
|
||||
// Keyed providers never auto-detect (the engine also forces this to null).
|
||||
detect() { return null; },
|
||||
|
||||
async fetch(entry, ctx) {
|
||||
const token = ctx?.env?.APIFY_TOKEN || process.env.APIFY_TOKEN;
|
||||
if (!hasToken(token)) {
|
||||
throw new Error('APIFY_TOKEN not set — enable apify in config/plugins.yml and add the token to .env');
|
||||
}
|
||||
if (!entry.actor) {
|
||||
throw new Error(`apify: entry ${entry.name} missing 'actor' (e.g. misceres/indeed-scraper)`);
|
||||
}
|
||||
if (
|
||||
!entry.field_map ||
|
||||
!isFieldSpec(entry.field_map.title) ||
|
||||
!isFieldSpec(entry.field_map.url) ||
|
||||
(entry.field_map.company != null && !isFieldSpec(entry.field_map.company)) ||
|
||||
(entry.field_map.location != null && !isFieldSpec(entry.field_map.location)) ||
|
||||
(entry.field_map.description != null && !isFieldSpec(entry.field_map.description))
|
||||
) {
|
||||
throw new Error(
|
||||
`apify: entry ${entry.name} has invalid field_map. Each of title, url, company, ` +
|
||||
`location, description must be a string or a non-empty array of strings. title and url are required.`
|
||||
);
|
||||
}
|
||||
|
||||
const opts = { token };
|
||||
if (entry.timeout_ms != null) opts.timeoutMs = entry.timeout_ms;
|
||||
const items = await runActor(entry.actor, entry.input || {}, opts);
|
||||
|
||||
const useLocalJd = entry.field_map.description != null;
|
||||
const sourceLabel = String(entry.actor || 'apify').replace(/[^a-z0-9]+/gi, '-').toLowerCase();
|
||||
|
||||
return items
|
||||
.map(item => {
|
||||
const normalized = normalizeItem(item, entry.field_map, entry.defaults);
|
||||
if (!normalized.title || !normalized.url) return null;
|
||||
if (!isHttpsUrl(normalized.url)) return null;
|
||||
if (!useLocalJd) return normalized;
|
||||
const descriptionBody = htmlToText(pickField(item, entry.field_map.description));
|
||||
if (!descriptionBody || descriptionBody.length < MIN_JD_BODY_CHARS) {
|
||||
return normalized;
|
||||
}
|
||||
const remoteUrl = normalized.url;
|
||||
const jdPath = saveJd(normalized, descriptionBody, sourceLabel);
|
||||
if (jdPath === null) return normalized;
|
||||
normalized.url = `local:${jdPath}`;
|
||||
normalized._remote_url = remoteUrl;
|
||||
return normalized;
|
||||
})
|
||||
.filter(j => j && j.title && j.url);
|
||||
},
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"id": "apify",
|
||||
"name": "Apify",
|
||||
"version": "1.0.0",
|
||||
"apiVersion": 1,
|
||||
"description": "Keyed job source: run an Apify actor and map its results into the scanner.",
|
||||
"hooks": ["provider"],
|
||||
"requiredEnv": ["APIFY_TOKEN"],
|
||||
"allowedHosts": ["api.apify.com"],
|
||||
"skill": "skill.md",
|
||||
"humanInTheLoop": true,
|
||||
"homepage": "https://apify.com"
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: career-ops-plugin-apify
|
||||
description: How to scan a job source through an Apify actor as a keyed provider.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# apify plugin
|
||||
|
||||
A keyed provider: runs an Apify actor and maps its dataset items into the
|
||||
scanner. It fires ONLY on a `portals.yml` entry that sets `provider: apify` —
|
||||
never via auto-detection. Put `APIFY_TOKEN` in `.env`.
|
||||
|
||||
## portals.yml entry
|
||||
|
||||
```yaml
|
||||
tracked_companies:
|
||||
- name: "Indeed — VP Engineering (Chicago)"
|
||||
provider: apify
|
||||
actor: misceres/indeed-scraper
|
||||
input: { position: "VP of Engineering", location: "Chicago, IL", maxItems: 25 }
|
||||
field_map:
|
||||
title: [positionName, title] # array = first non-empty wins
|
||||
url: url
|
||||
company: [company, companyName]
|
||||
location: [location, formattedLocation]
|
||||
```
|
||||
|
||||
## Then
|
||||
|
||||
`node scan.mjs` runs the provider for that entry and writes the results to the
|
||||
pipeline like any other source. An optional `field_map.description` caches the
|
||||
JD locally under `jds/`.
|
||||
@@ -0,0 +1,115 @@
|
||||
// @ts-check
|
||||
// Pure, side-effect-free Gmail helpers. Ported verbatim from the gmail-helpers
|
||||
// contributed by @SparshGarg999 in #1203 (with thanks). Files prefixed with _
|
||||
// are never discovered as plugins.
|
||||
|
||||
/**
|
||||
* Extract all http/https URLs from a string (plain text or HTML). Normalizes
|
||||
* & and strips trailing punctuation. Dedups.
|
||||
* @param {string} body
|
||||
* @returns {string[]}
|
||||
*/
|
||||
export function extractUrls(body) {
|
||||
if (!body) return [];
|
||||
const urls = [];
|
||||
const regex = /https?:\/\/[^\s"'<>\(\)]+/gi;
|
||||
let match;
|
||||
while ((match = regex.exec(body)) !== null) {
|
||||
const url = match[0].replace(/[.,;:!?]+$/, '').replace(/&/g, '&');
|
||||
urls.push(url);
|
||||
}
|
||||
return [...new Set(urls)];
|
||||
}
|
||||
|
||||
/**
|
||||
* Is a URL clean and relevant (not a click tracker, unsubscribe link, or pixel)?
|
||||
* @param {string} url
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function isCleanUrl(url) {
|
||||
try {
|
||||
const u = new URL(url);
|
||||
const lowerUrl = url.toLowerCase();
|
||||
const badKeywords = [
|
||||
'click', 'track', 'openpixel', 'sendgrid', 'unsubscribe', 'optout',
|
||||
'newsletter', 'subscribe', 'w3.org', 'doubleclick', 'googlesyndication',
|
||||
'googleadservices', 'mailgun', 'mandrill', 'mjml', 'github.com/login',
|
||||
'linkedin.com/legal', 'linkedin.com/help', 'linkedin.com/settings',
|
||||
];
|
||||
if (badKeywords.some(kw => lowerUrl.includes(kw))) return false;
|
||||
return u.protocol === 'https:';
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* DMARC alignment check (anti-spoof gate, fail-closed). Only emails whose
|
||||
* Authentication-Results header reports dmarc=pass are trusted.
|
||||
* @param {Array<{ name: string, value: string }>} headers
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function isAuthenticEmail(headers) {
|
||||
if (!Array.isArray(headers)) return false;
|
||||
for (const h of headers) {
|
||||
if (h.name && h.name.toLowerCase() === 'authentication-results') {
|
||||
if (h.value && /dmarc=pass/i.test(h.value)) return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse "{Role} at {Company}" from a subject line.
|
||||
* @param {string} subject
|
||||
* @returns {{ role: string, company: string } | null}
|
||||
*/
|
||||
export function parseRoleAtCompany(subject) {
|
||||
if (!subject) return null;
|
||||
let clean = subject.replace(/^(re|fwd|new match|job alert|alert|match|notification|alert for|daily alert for):\s*/i, '').trim();
|
||||
clean = clean.split(/\s+[-|]\s+/)[0].trim();
|
||||
const match = clean.match(/^(.+?)\s+at\s+(.+)$/i);
|
||||
if (match) {
|
||||
const role = match[1].trim();
|
||||
const company = match[2].trim();
|
||||
if (role && company && role.length < 100 && company.length < 100) {
|
||||
return { role, company };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively decode a Gmail message payload's base64url body parts to text.
|
||||
* @param {any} payload
|
||||
* @returns {string}
|
||||
*/
|
||||
export function getMessageBody(payload) {
|
||||
if (!payload) return '';
|
||||
let body = '';
|
||||
if (payload.body && payload.body.data) {
|
||||
const base64 = payload.body.data.replace(/-/g, '+').replace(/_/g, '/');
|
||||
body += Buffer.from(base64, 'base64').toString('utf-8');
|
||||
}
|
||||
if (payload.parts) {
|
||||
for (const part of payload.parts) body += getMessageBody(part);
|
||||
}
|
||||
return body;
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort company name from a known ATS URL (greenhouse/lever slug).
|
||||
* @param {string} url
|
||||
* @returns {string}
|
||||
*/
|
||||
export function companyFromUrl(url) {
|
||||
try {
|
||||
const { hostname, pathname } = new URL(url);
|
||||
if (hostname === 'boards.greenhouse.io' || hostname.endsWith('.greenhouse.io') ||
|
||||
hostname === 'jobs.lever.co' || hostname.endsWith('.lever.co')) {
|
||||
const parts = pathname.split('/').filter(Boolean);
|
||||
if (parts.length > 0) return parts[0];
|
||||
}
|
||||
} catch { /* malformed → no company */ }
|
||||
return '';
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
// @ts-check
|
||||
// ── Reference seed ── This bundled plugin is a stable, reviewed example. To
|
||||
// extend it, publish career-ops-plugin-<id> with "supersedesBundled": true and
|
||||
// your version takes precedence once installed (see docs/PLUGINS.md). Bundled
|
||||
// seeds take only security/compat fixes — feature work happens in the successor repo.
|
||||
//
|
||||
// Gmail ingest plugin — pulls job leads from a Gmail label into your pipeline.
|
||||
//
|
||||
// Ported from the email-driven ingestion contributed by @SparshGarg999 in #1203
|
||||
// (with thanks), reshaped to the plugin contract: OAuth credentials come from
|
||||
// the scoped ctx.env (not credential files), the label/days_back come from
|
||||
// ctx.settings (config/plugins.yml), and the hook RETURNS Job[] — the engine
|
||||
// (plugins.mjs), not this plugin, writes them to pipeline.md canonically. No
|
||||
// `search://` rows are emitted (they aren't real URLs the pipeline can open).
|
||||
//
|
||||
// Enable in config/plugins.yml:
|
||||
// gmail: { enabled: true, label: "Job Leads", days_back: 7 }
|
||||
// Add to .env: GMAIL_CLIENT_ID, GMAIL_CLIENT_SECRET, GMAIL_REFRESH_TOKEN.
|
||||
// Run: node plugins.mjs run gmail
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import {
|
||||
extractUrls, isCleanUrl, isAuthenticEmail, parseRoleAtCompany,
|
||||
getMessageBody, companyFromUrl,
|
||||
} from './_helpers.mjs';
|
||||
|
||||
const TOKEN_URL = 'https://oauth2.googleapis.com/token';
|
||||
const GMAIL_API = 'https://gmail.googleapis.com/gmail/v1/users/me';
|
||||
const STATE_PATH = 'data/gmail-state.json'; // the plugin's own processed-id cursor
|
||||
|
||||
/** Exchange the long-lived refresh token for a short-lived access token. */
|
||||
async function getAccessToken({ clientId, clientSecret, refreshToken }, fetchFn = globalThis.fetch) {
|
||||
const res = await fetchFn(TOKEN_URL, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
||||
body: new URLSearchParams({
|
||||
client_id: clientId,
|
||||
client_secret: clientSecret,
|
||||
refresh_token: refreshToken,
|
||||
grant_type: 'refresh_token',
|
||||
}),
|
||||
});
|
||||
if (!res.ok) {
|
||||
throw new Error(`Gmail token refresh failed: ${res.status} ${(await res.text()).slice(0, 200)}`);
|
||||
}
|
||||
const data = await res.json();
|
||||
if (!data.access_token) throw new Error('Gmail token refresh returned no access_token');
|
||||
return data.access_token;
|
||||
}
|
||||
|
||||
function loadProcessedIds() {
|
||||
if (!existsSync(STATE_PATH)) return new Set();
|
||||
try {
|
||||
const state = JSON.parse(readFileSync(STATE_PATH, 'utf-8'));
|
||||
return new Set(state.processed_message_ids || []);
|
||||
} catch {
|
||||
return new Set();
|
||||
}
|
||||
}
|
||||
|
||||
function saveProcessedIds(ids) {
|
||||
try {
|
||||
mkdirSync('data', { recursive: true });
|
||||
writeFileSync(STATE_PATH, JSON.stringify({ processed_message_ids: [...ids] }, null, 2), 'utf-8');
|
||||
} catch (err) {
|
||||
console.warn(`gmail: could not persist processed-id state — ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/** @type {{ ingest: (ctx: any) => Promise<object[]> }} */
|
||||
export default {
|
||||
async ingest(ctx) {
|
||||
const clientId = ctx?.env?.GMAIL_CLIENT_ID;
|
||||
const clientSecret = ctx?.env?.GMAIL_CLIENT_SECRET;
|
||||
const refreshToken = ctx?.env?.GMAIL_REFRESH_TOKEN;
|
||||
if (!clientId || !clientSecret || !refreshToken) {
|
||||
throw new Error('gmail: missing GMAIL_CLIENT_ID / GMAIL_CLIENT_SECRET / GMAIL_REFRESH_TOKEN in .env');
|
||||
}
|
||||
|
||||
const label = ctx?.settings?.label || 'Job Leads';
|
||||
const daysBack = Number(ctx?.settings?.days_back ?? 7);
|
||||
if (!Number.isInteger(daysBack) || daysBack <= 0) {
|
||||
throw new Error(`gmail: invalid days_back "${ctx?.settings?.days_back}" (must be a positive integer)`);
|
||||
}
|
||||
|
||||
const token = await getAccessToken({ clientId, clientSecret, refreshToken }, ctx.fetch);
|
||||
const auth = { Authorization: `Bearer ${token}` };
|
||||
const query = `label:"${label}" newer_than:${daysBack}d`;
|
||||
ctx.log(`gmail: querying ${query}`);
|
||||
|
||||
// List message ids (paginated). ctx.fetch throws on a non-2xx (with the body
|
||||
// in the message), so a failed page surfaces a clear error.
|
||||
const messages = [];
|
||||
let pageToken = null;
|
||||
do {
|
||||
let url = `${GMAIL_API}/messages?q=${encodeURIComponent(query)}`;
|
||||
if (pageToken) url += `&pageToken=${pageToken}`;
|
||||
const data = await (await ctx.fetch(url, { headers: auth })).json();
|
||||
if (data.messages) messages.push(...data.messages);
|
||||
pageToken = data.nextPageToken;
|
||||
} while (pageToken);
|
||||
|
||||
const processedIds = loadProcessedIds();
|
||||
const seenUrls = new Set();
|
||||
const jobs = [];
|
||||
|
||||
for (const m of messages) {
|
||||
if (processedIds.has(m.id)) continue;
|
||||
// Per-message resilience: a single bad detail fetch is skipped, not fatal.
|
||||
let msg;
|
||||
try {
|
||||
msg = await (await ctx.fetch(`${GMAIL_API}/messages/${m.id}?format=full`, { headers: auth })).json();
|
||||
} catch (err) {
|
||||
console.warn(`gmail: failed to fetch message ${m.id} — ${err.message}`);
|
||||
continue;
|
||||
}
|
||||
const headers = msg.payload?.headers || [];
|
||||
const subject = headers.find(h => h.name?.toLowerCase() === 'subject')?.value || '';
|
||||
|
||||
// Fail-closed on spoofed mail (DMARC).
|
||||
if (!isAuthenticEmail(headers)) {
|
||||
console.warn(`gmail: skipping spoofed/unauthenticated email "${subject}"`);
|
||||
processedIds.add(m.id);
|
||||
continue;
|
||||
}
|
||||
|
||||
const seed = parseRoleAtCompany(subject);
|
||||
const cleanUrls = extractUrls(getMessageBody(msg.payload)).filter(isCleanUrl);
|
||||
for (const url of cleanUrls) {
|
||||
if (seenUrls.has(url)) continue;
|
||||
seenUrls.add(url);
|
||||
jobs.push({
|
||||
title: seed?.role || 'Job lead (email)',
|
||||
url,
|
||||
company: companyFromUrl(url) || seed?.company || '',
|
||||
location: '',
|
||||
});
|
||||
}
|
||||
processedIds.add(m.id);
|
||||
}
|
||||
|
||||
saveProcessedIds(processedIds);
|
||||
return jobs;
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"id": "gmail",
|
||||
"name": "Gmail ingest",
|
||||
"version": "1.0.0",
|
||||
"apiVersion": 1,
|
||||
"description": "Pull job leads from a Gmail label into your pipeline (read-only, your OAuth token).",
|
||||
"hooks": ["ingest"],
|
||||
"requiredEnv": ["GMAIL_CLIENT_ID", "GMAIL_CLIENT_SECRET", "GMAIL_REFRESH_TOKEN"],
|
||||
"allowedHosts": ["oauth2.googleapis.com", "gmail.googleapis.com"],
|
||||
"skill": "skill.md",
|
||||
"humanInTheLoop": true,
|
||||
"homepage": "https://developers.google.com/gmail/api"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: career-ops-plugin-gmail
|
||||
description: How to pull job leads from a Gmail label into the career-ops pipeline.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# gmail plugin
|
||||
|
||||
Reads a Gmail label, extracts clean job URLs from authentic (DMARC-passing)
|
||||
emails, and returns them as leads. The engine writes them to the pipeline.
|
||||
|
||||
## Command
|
||||
|
||||
- `node plugins.mjs run gmail` — ingest new leads from the configured label.
|
||||
|
||||
## Setup
|
||||
|
||||
Put `GMAIL_CLIENT_ID` + `GMAIL_CLIENT_SECRET` + `GMAIL_REFRESH_TOKEN` in `.env`
|
||||
(an OAuth Desktop client + a refresh token from the consent flow). Configure the
|
||||
label + lookback in `config/plugins.yml`:
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
gmail: { enabled: true, label: "Job Leads", days_back: 7 }
|
||||
```
|
||||
|
||||
## Data it produces
|
||||
|
||||
`Job[]` ({ title, url, company, location }) — the engine de-dups against the
|
||||
pipeline and appends new ones. It maintains its own processed-message cursor in
|
||||
`data/gmail-state.json` to avoid re-reading the same emails.
|
||||
@@ -0,0 +1,137 @@
|
||||
// @ts-check
|
||||
// Notion API helper for the notion plugin. Ported from the notion-lib.mjs
|
||||
// contributed by @pcomans in #959 (with thanks), reshaped for the plugin layer:
|
||||
//
|
||||
// - No module-level secrets / `import 'dotenv/config'`. The token + parent page
|
||||
// id are passed into createNotionClient() from the plugin's scoped ctx.env,
|
||||
// so nothing reads process.env at import time and the engine stays the place
|
||||
// that decides when .env is loaded.
|
||||
// - templates/states.yml (the canonical status source of truth) is resolved
|
||||
// from the repo root, two levels up from this bundled plugin.
|
||||
//
|
||||
// Pages use Notion's native markdown on both sides. The 360ms inter-request
|
||||
// sleep keeps us under ~3 req/s. DB resolution is by NAME so no workspace id is
|
||||
// ever embedded. Files prefixed with _ are never discovered as plugins.
|
||||
|
||||
import { readFileSync } from 'fs';
|
||||
import { dirname, join } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import yaml from 'js-yaml';
|
||||
|
||||
const DIR = dirname(fileURLToPath(import.meta.url));
|
||||
const STATES_PATH = join(DIR, '..', '..', 'templates', 'states.yml');
|
||||
const MAX = 1900; // safety margin under Notion's 2000-char rich_text limit
|
||||
|
||||
// ── canonical states (templates/states.yml is the source of truth) ──────────
|
||||
let _states;
|
||||
function loadStates() {
|
||||
if (_states) return _states;
|
||||
const doc = yaml.load(readFileSync(STATES_PATH, 'utf-8'));
|
||||
const labels = [], aliasMap = {};
|
||||
for (const s of doc.states) {
|
||||
labels.push(s.label);
|
||||
aliasMap[s.label.toLowerCase()] = s.label;
|
||||
for (const a of (s.aliases || [])) aliasMap[String(a).toLowerCase()] = s.label;
|
||||
}
|
||||
_states = { labels, aliasMap };
|
||||
return _states;
|
||||
}
|
||||
/** Canonical label for a status (case-insensitive, alias-aware), or null. */
|
||||
export function canonicalStatus(raw) {
|
||||
if (!raw) return null;
|
||||
const key = String(raw).replace(/\*\*/g, '').trim().toLowerCase();
|
||||
return loadStates().aliasMap[key] || null;
|
||||
}
|
||||
export function statusLabels() { return loadStates().labels.slice(); }
|
||||
|
||||
// ── text → rich_text for property values (chunk-safe under the 2000-char limit) ──
|
||||
export function rich(text) {
|
||||
const str = String(text ?? '');
|
||||
const out = [];
|
||||
for (let i = 0; i < str.length || out.length === 0; i += MAX) out.push({ type: 'text', text: { content: str.slice(i, i + MAX) } });
|
||||
return out;
|
||||
}
|
||||
|
||||
export function plain(prop) {
|
||||
return (prop?.title || prop?.rich_text || []).map((t) => t.plain_text).join('');
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a Notion client bound to one user's token + parent page. Network goes
|
||||
* through the injected `fetchFn` (the plugin passes ctx.fetch so the engine's
|
||||
* allowedHosts/HTTPS/redirect guard applies); falls back to global fetch for
|
||||
* standalone use. Nothing here reads process.env.
|
||||
* @param {{ token: string, parent: string, fetch?: Function }} cfg
|
||||
*/
|
||||
export function createNotionClient({ token, parent, fetch: fetchFn = globalThis.fetch }) {
|
||||
if (!token) throw new Error('NOTION_ACCESS_TOKEN is not set (.env) — the Notion plugin needs it to read/write.');
|
||||
const HEADERS = { Authorization: `Bearer ${token}`, 'Notion-Version': '2025-09-03', 'Content-Type': 'application/json' };
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
async function api(path, method, body) {
|
||||
await sleep(360); // ~3 req/s
|
||||
// ctx.fetch throws on non-2xx (its message carries the body); the !r.ok
|
||||
// branch below is the fallback when a plain global fetch is injected.
|
||||
const r = await fetchFn(`https://api.notion.com/v1/${path}`, { method, headers: HEADERS, body: body ? JSON.stringify(body) : undefined });
|
||||
const j = await r.json();
|
||||
if (!r.ok) throw new Error(`Notion ${method} ${path} -> ${j.code}: ${j.message}`);
|
||||
return j;
|
||||
}
|
||||
|
||||
/** Create a page in a data source. `markdown` (optional) becomes the page body. */
|
||||
async function createPage(dataSourceId, properties, markdown) {
|
||||
const body = { parent: { type: 'data_source_id', data_source_id: dataSourceId }, properties };
|
||||
if (markdown) body.markdown = markdown;
|
||||
return api('pages', 'POST', body);
|
||||
}
|
||||
|
||||
/** Map of DB name → primary data source id for every DB under the parent page. */
|
||||
async function resolveDBs() {
|
||||
if (!parent) throw new Error('Set NOTION_PARENT_PAGE_ID in .env (the "Career Ops" parent page id).');
|
||||
const out = {};
|
||||
let cursor;
|
||||
do {
|
||||
const j = await api(`blocks/${parent}/children?page_size=100${cursor ? `&start_cursor=${cursor}` : ''}`, 'GET');
|
||||
for (const b of j.results) {
|
||||
if (b.type !== 'child_database') continue;
|
||||
const db = await api(`databases/${b.id}`, 'GET');
|
||||
out[b.child_database.title] = db.data_sources?.[0]?.id;
|
||||
}
|
||||
cursor = j.has_more ? j.next_cursor : null;
|
||||
} while (cursor);
|
||||
return out;
|
||||
}
|
||||
|
||||
async function queryDB(dataSourceId) {
|
||||
let cursor, all = [];
|
||||
do {
|
||||
const j = await api(`data_sources/${dataSourceId}/query`, 'POST', { page_size: 100, start_cursor: cursor });
|
||||
all.push(...j.results);
|
||||
cursor = j.has_more ? j.next_cursor : null;
|
||||
} while (cursor);
|
||||
return all;
|
||||
}
|
||||
|
||||
function summarize(r) {
|
||||
return {
|
||||
id: r.id,
|
||||
company: plain(r.properties.Company),
|
||||
role: plain(r.properties.Role),
|
||||
status: r.properties.Status?.select?.name || '',
|
||||
score: r.properties.Score?.number ?? null,
|
||||
jobUrl: r.properties.URL?.url || '',
|
||||
url: r.url,
|
||||
};
|
||||
}
|
||||
|
||||
/** Match against "<company> / <role>" (substring, case-insensitive) or exact company. */
|
||||
async function findRecords(dataSourceId, match) {
|
||||
const m = String(match).toLowerCase().trim();
|
||||
return (await queryDB(dataSourceId)).map(summarize).filter((r) => {
|
||||
const hay = `${r.company} / ${r.role}`.toLowerCase();
|
||||
return hay.includes(m) || r.company.toLowerCase() === m;
|
||||
});
|
||||
}
|
||||
|
||||
return { api, createPage, resolveDBs, queryDB, findRecords };
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
// @ts-check
|
||||
// ── Reference seed ── This bundled plugin is a stable, reviewed example. To
|
||||
// extend it, publish career-ops-plugin-<id> with "supersedesBundled": true and
|
||||
// your version takes precedence once installed (see docs/PLUGINS.md). Bundled
|
||||
// seeds take only security/compat fixes — feature work happens in the successor repo.
|
||||
//
|
||||
// Notion plugin — mirror your tracker to a Notion database (export) and read
|
||||
// records back as job leads (search).
|
||||
//
|
||||
// Built on the Notion backend contributed by @pcomans in #959 (with thanks),
|
||||
// reshaped per the plugin contract. The decisive change: Notion is an OPT-IN
|
||||
// MIRROR, not a replacement backend. data/applications.md stays the canonical
|
||||
// source of truth (the web reads it); `export` pushes a read-only snapshot of it
|
||||
// to the user's own Notion DB. The core never writes to Notion as primary, and
|
||||
// modes are not edited — this lives entirely behind `node plugins.mjs run notion`.
|
||||
//
|
||||
// Setup: a "Career Ops" parent page in Notion containing an "Applications" DB
|
||||
// with Company / Role / Status / Score / URL properties, shared with your
|
||||
// internal integration. Enable in config/plugins.yml; keys in .env.
|
||||
//
|
||||
// node plugins.mjs run notion export # mirror tracker → Notion
|
||||
// node plugins.mjs run notion search "platform" # read matching records → pipeline
|
||||
|
||||
import { createNotionClient, rich, canonicalStatus } from './_notion.mjs';
|
||||
|
||||
function clientFromCtx(ctx) {
|
||||
return createNotionClient({
|
||||
token: ctx?.env?.NOTION_ACCESS_TOKEN,
|
||||
parent: ctx?.env?.NOTION_PARENT_PAGE_ID,
|
||||
fetch: ctx?.fetch, // route through the engine's allowedHosts/redirect guard
|
||||
});
|
||||
}
|
||||
|
||||
async function applicationsDb(client) {
|
||||
const dbs = await client.resolveDBs();
|
||||
const apps = dbs['Applications'];
|
||||
if (!apps) throw new Error('No "Applications" database found under the Career Ops page — create it and share the integration with it.');
|
||||
return apps;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a tracker score cell into a numeric value for the Notion DB Score property.
|
||||
*
|
||||
* Scores in applications.md may be formatted like `4.2/5`, `**4.2/5**`, `4.25`, etc.
|
||||
* Strips formatting and extracts the first numeric value so slash-formatted
|
||||
* scores (e.g. 4.2/5) are not mangled into 4.25 (#1414).
|
||||
*
|
||||
* @param {unknown} s - Raw score value from tracker row.
|
||||
* @returns {number} Parsed score, or NaN if no valid number is present.
|
||||
*/
|
||||
export function parseScore(s) {
|
||||
const m = String(s ?? '').replace(/\*\*/g, '').match(/([\d.]+)/);
|
||||
return m ? parseFloat(m[1]) : NaN;
|
||||
}
|
||||
|
||||
export default {
|
||||
parseScore,
|
||||
|
||||
/**
|
||||
* export: upsert each tracker row into the user's Notion Applications DB.
|
||||
* Receives a frozen read-only snapshot of the tracker — never a file handle.
|
||||
* @param {{ applications: Array<Record<string,string>> }} snapshot
|
||||
* @param {any} ctx
|
||||
*/
|
||||
async export(snapshot, ctx) {
|
||||
const rows = Array.isArray(snapshot?.applications) ? snapshot.applications : [];
|
||||
if (rows.length === 0) return { pushed: 0 };
|
||||
const client = clientFromCtx(ctx);
|
||||
const apps = await applicationsDb(client);
|
||||
|
||||
let pushed = 0;
|
||||
for (const row of rows) {
|
||||
const company = (row.company || '').trim();
|
||||
const role = (row.role || '').trim();
|
||||
if (!company || !role) continue;
|
||||
|
||||
const props = { Role: { title: rich(role) }, Company: { rich_text: rich(company) } };
|
||||
const status = canonicalStatus(row.status);
|
||||
if (status) props.Status = { select: { name: status } };
|
||||
const score = parseScore(row.score);
|
||||
if (Number.isFinite(score)) props.Score = { number: score };
|
||||
|
||||
if (ctx?.dryRun) { ctx.log(`would push: ${company} — ${role}`); pushed++; continue; }
|
||||
|
||||
// Upsert: update an existing company+role record, else create one.
|
||||
const dupes = (await client.findRecords(apps, `${company} / ${role}`))
|
||||
.filter((h) => h.company.toLowerCase() === company.toLowerCase() && h.role.toLowerCase() === role.toLowerCase());
|
||||
if (dupes.length) await client.api(`pages/${dupes[0].id}`, 'PATCH', { properties: props });
|
||||
else await client.createPage(apps, props);
|
||||
pushed++;
|
||||
}
|
||||
return { pushed };
|
||||
},
|
||||
|
||||
/**
|
||||
* search: return Notion records matching a query as Job[]. Only records that
|
||||
* carry a job posting in a `URL` property are returned (e.g. a postings DB, or
|
||||
* leads you added in Notion). Note: `export` mirrors the tracker (company/role/
|
||||
* status/score) and does NOT set a job URL, so export-created rows are not
|
||||
* round-tripped by search — that's intentional, they already live in your
|
||||
* tracker. The engine writes any results to the pipeline canonically.
|
||||
* @param {string} query
|
||||
* @param {any} ctx
|
||||
*/
|
||||
async search(query, ctx) {
|
||||
const client = clientFromCtx(ctx);
|
||||
const apps = await applicationsDb(client);
|
||||
const hits = await client.findRecords(apps, query);
|
||||
return hits
|
||||
.filter((h) => h.jobUrl && /^https?:\/\//i.test(h.jobUrl))
|
||||
.map((h) => ({ title: h.role || 'Notion record', url: h.jobUrl, company: h.company || '', location: '' }));
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"id": "notion",
|
||||
"name": "Notion",
|
||||
"version": "1.0.0",
|
||||
"apiVersion": 1,
|
||||
"description": "Mirror your tracker to a Notion database and read records back as job leads.",
|
||||
"hooks": ["export", "search"],
|
||||
"requiredEnv": ["NOTION_ACCESS_TOKEN", "NOTION_PARENT_PAGE_ID"],
|
||||
"allowedHosts": ["api.notion.com"],
|
||||
"skill": "skill.md",
|
||||
"humanInTheLoop": true,
|
||||
"homepage": "https://developers.notion.com"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: career-ops-plugin-notion
|
||||
description: How to mirror the career-ops tracker to a Notion database and read records back as job leads.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# notion plugin
|
||||
|
||||
Mirrors your application tracker to a Notion database (export) and reads records
|
||||
back into the pipeline (search). `data/applications.md` stays the source of
|
||||
truth — Notion is an additive mirror.
|
||||
|
||||
## Commands
|
||||
|
||||
- `node plugins.mjs run notion export` — push each tracker row (company / role /
|
||||
status / score) to the "Applications" database under your Career Ops page.
|
||||
Add `--dry-run` to preview without writing.
|
||||
- `node plugins.mjs run notion search "<query>"` — return Notion records that
|
||||
carry a job URL, matching the query, and append them to the pipeline.
|
||||
|
||||
## Setup
|
||||
|
||||
A "Career Ops" parent page in Notion containing an "Applications" database with
|
||||
Company / Role / Status / Score / URL properties, shared with your internal
|
||||
integration. Put `NOTION_ACCESS_TOKEN` + `NOTION_PARENT_PAGE_ID` in `.env`.
|
||||
|
||||
## Data it produces
|
||||
|
||||
`search` returns `Job[]` ({ title, url, company, location }) for records that
|
||||
have a job URL; the engine writes them to the pipeline. `export` returns
|
||||
{ pushed: N } — it never writes local files.
|
||||
Reference in new issue
Block a user