Skip to content

Plugin framework

The existing plugin framework allows adopters to publish npm or PyPI packages that declare custom fields for CommonGrants schema objects. We want to expand this framework to also support bidirectional data transformations — toCommon/fromCommon functions that convert between a source system’s native data shape and the CommonGrants schema. We also want to organize this framework to support other potential future features with minimal rework.

Plugin authors are free to implement toCommon/fromCommon as plain hand-written functions. The SDK also provides a buildTransforms() / build_transforms() utility (see ADR-0017) that can generate these functions from separate declarative mapping objects, but using it is not required.

How should the Plugin object be structured to support both custom field declarations and bidirectional transforms, while enabling clean dependency injection and remaining stable as the protocol’s object list grows?

  • Should the top-level Plugin structure group by feature (meta, client, schemas, extensions) or by object (Opportunity, Application, …)?
  • Should client configuration (auth, transport, rate-limiting) sit alongside per-object schemas, or be lifted to the top level as a system-level concern?
  • Should custom fields and transforms be coupled in the same package, or allowed independently?
  • The framework must be implementable in both the Python and TypeScript SDKs with as consistent an interface as possible.
  • The extensions config must be serializable and able to pass validation (JSON-safe), and must be combinable across multiple extension packages via mergeExtensions() (TypeScript) / merge_extensions() (Python).
  • Existing plugin packages that declare only custom fields should remain valid with minimal changes.
  • The SDK interface should support clean dependency injection — it must be possible to pass client or schemas as a coherent unit without reassembling them from per-object branches.
  • Auth, transport, and rate-limiting are system-level concerns that belong to a single client, not distributed across per-object branches.
  • The top-level Plugin surface should be short and stable — adding new protocol objects should not expand the top-level key set.

We decided to:

  1. Keep “plugin” as the unified term for both the published npm/PyPI packages in the website catalog and the runtime SDK object. No change to PluginSourceEntry or src/content/plugins/index.json. The existing definePlugin() function is expanded to accept the full set of top-level fields described below.

  2. Use functional grouping at the top level with four keys — meta, client, schemas, and extensions — rather than grouping by object name at the root.

  3. Use per-object grouping inside schemas where it reflects real coupling: each object’s native schema, CommonGrants schema, and bidirectional transforms are tightly coupled and change together.

  4. Expand definePlugin() to accept all top-level fieldsmeta, client, schemas, and extensions — rather than only extensions. extensions will contain any parts of the plugin that can be merged with other plugins by using mergeExtensions(), which has a flag for handling key conflicts.

  5. Make all top-level Plugin fields optional so adopters can publish a plugin that provides only the features they need — for example, custom fields only — and expand to include transforms, client config, or additional schemas incrementally over time.

The resulting Plugin shape:

plugin.meta // name, version, sourceSystem, capabilities
plugin.getClient // (config: ClientConfig) => Client; memoized by definePlugin()
plugin.extensions // serializable; used by mergeExtensions()
plugin.schemas.<ObjectName> = { native, common, toCommon, fromCommon }
type PluginCapability =
| "customFields" // declares custom fields on CommonGrants schema objects
| "customFilters" // declares custom filter parameters for resource methods
| "transforms" // provides toCommon/fromCommon transformation functions
| "client"; // provides a runtime client (auth, transport, resource methods)
interface PluginMeta {
name: string;
version?: string; // optional; if omitted, definePlugin() infers it from the package's package.json
sourceSystem: string;
capabilities?: PluginCapability[];
}
// Defined in lib/ts-sdk/src/extensions/types.ts — reproduced here for reference
interface CustomFieldSpec {
name?: string; // optional; dict key is used as the display name fallback
fieldType: CustomFieldType; // enum defined in the SDK
value?: z.ZodTypeAny; // optional Zod schema to validate the value; defaults based on fieldType
description?: string;
}
// Runtime type — produced by definePlugin(), not provided directly by plugin authors
interface ObjectSchemas<TNative, TCommon> {
native: ZodType<TNative>;
common: ZodType<TCommon>;
toCommon: (native: TNative) => TCommon;
fromCommon: (common: TCommon) => TNative;
}
// Input type — provided by plugin authors inside DefinePluginOptions.schemas
interface ObjectSchemasInput<TNative = unknown, TCommon = unknown> {
native?: ZodType<TNative>; // defaults to Record<string, unknown> if omitted
toCommon?: (native: TNative) => TCommon;
fromCommon?: (common: TCommon) => TNative;
}
// Scalar types only — filters are query parameters, not schema fields
type CustomFilterType = "string" | "number" | "integer" | "boolean";
interface CustomFilterSpec {
filterType: CustomFilterType;
description?: string;
}
// Per-object config shape inside extensions.schemas — mirrors Python's PluginExtensionsSchema
interface PluginExtensionsObjectConfig {
customFields?: Record<string, CustomFieldSpec>;
}
// Serializable portion of the plugin config — safe to store as JSON.
// schemas keys are restricted to ExtensibleSchemaName (the known set of CommonGrants
// objects that support custom fields), following the existing SchemaExtensions pattern.
interface PluginExtensions {
meta?: Partial<PluginMeta>;
schemas?: Partial<Record<ExtensibleSchemaName, PluginExtensionsObjectConfig>>;
}
// ClientConfig is defined by the plugin author to declare the system-specific inputs
// they require (e.g. auth token, base URL, max page size, timeouts).
interface ClientConfig {
[key: string]: unknown;
}
// Client is a placeholder for the SDK's runtime client type (not shown here)
interface Plugin {
meta?: PluginMeta;
getClient?: (config: ClientConfig) => Client;
extensions?: PluginExtensions; // serializable
schemas?: Partial<
Record<ExtensibleSchemaName, ObjectSchemas<unknown, unknown>>
>;
filters?: Partial<
Record<ExtensibleSchemaName, Record<string, CustomFilterSpec>>
>;
}
// Input object for definePlugin(). Using a named-options object makes it easy to add
// new inputs over time without breaking existing callers.
interface DefinePluginOptions {
meta?: PluginMeta;
// Plugin authors provide a factory function; definePlugin() wraps it with memoization
// so the same Client instance is returned for equivalent configs automatically.
getClient?: (config: ClientConfig & { auth?: AuthMethod }) => Client;
extensions?: PluginExtensions; // serializable
// Plugin authors provide input schemas and transforms; definePlugin() compiles them
// into the full ObjectSchemas runtime type, merging any customFields from extensions.
schemas?: Partial<Record<ExtensibleSchemaName, ObjectSchemasInput>>;
filters?: Partial<
Record<ExtensibleSchemaName, Record<string, CustomFilterSpec>>
>;
}
// Factory: all options are optional so adopters can start with only what they need
// and expand incrementally.
//
// definePlugin compiles DefinePluginOptions into a Plugin by:
// - extending the base CommonGrants schema with any declared customFields → common
// - native defaults to Record<string, unknown> if omitted (extensions is JSON-safe;
// runtime Zod schemas cannot be included)
// - wrapping getClient with memoization so the same Client instance is returned
// for equivalent configs automatically
//
// toCommon / fromCommon may be plain hand-written functions or generated via
// buildTransforms(). Either way they are provided in schemas, not derived from extensions.
function definePlugin(options: DefinePluginOptions): Plugin;
// Combine multiple extension objects (e.g. from separate packages)
function mergeExtensions(
...extensions: PluginExtensions[],
onConflict: "firstWins" = "firstWins",
): PluginExtensions;
// Utility: generates toCommon and fromCommon functions from separate declarative
// mapping objects (ADR-0017 format). Using this utility is optional — plugin authors
// may provide plain hand-written functions instead.
function buildTransforms<TNative, TCommon>(
toCommonMapping: Record<string, unknown>, // ADR-0017 mapping from native → CommonGrants
fromCommonMapping: Record<string, unknown>, // ADR-0017 mapping from CommonGrants → native
): {
toCommon: (native: TNative) => TCommon;
fromCommon: (common: TCommon) => TNative;
};
// buildTransforms() is a utility that generates toCommon/fromCommon from declarative
// mappings (ADR-0017). Using it is optional — plain functions work just as well.
const { toCommon, fromCommon } = buildTransforms(
// toCommon: native grants.gov shape → CommonGrants Opportunity
{
title: "data.opportunity_title",
status: {
value: {
match: {
field: "data.opportunity_status",
case: { posted: "open", archived: "closed" },
default: "custom",
},
},
},
},
// fromCommon: CommonGrants Opportunity → native grants.gov shape
{
"data.opportunity_title": "title",
"data.opportunity_status": {
value: {
match: {
field: "status",
case: { open: "posted", closed: "archived" },
default: "custom",
},
},
},
},
);
const plugin = definePlugin({
meta: {
name: "grants-gov-plugin",
version: "1.0.0",
sourceSystem: "grants.gov",
},
// definePlugin memoizes getClient — the same Client is returned for equivalent configs.
getClient: (config: ClientConfig & { auth?: AuthMethod }) =>
new Client({
baseUrl: config.baseUrl ?? "https://api.grants.gov",
timeout: config.timeout,
pageSize: config.pageSize,
maxItems: config.maxItems,
auth: config.auth,
}),
extensions: {
schemas: {
Opportunity: {
customFields: {
programArea: {
fieldType: CustomFieldType.String,
description: "HHS program area code",
},
legacyGrantId: {
fieldType: CustomFieldType.Integer,
description: "Numeric ID from the legacy grants system",
},
},
},
},
},
schemas: {
Opportunity: {
native: GrantsGovOpportunitySchema,
toCommon,
fromCommon,
},
},
});
// Combine extensions from multiple packages before constructing the plugin
const merged = mergeExtensions(baseExtensions, grantsGovExtensions);
const mergedPlugin = definePlugin({ extensions: merged });
// Calling getClient() with a config object — memoized, so repeated calls return the same instance
const client = plugin.getClient({ auth: Auth.bearer("token"), pageSize: 50 });
import { grantsGovPlugin } from "grants-gov-plugin";
// Get a configured client for this source system
const client = grantsGovPlugin.getClient({
auth: Auth.bearer(process.env.GRANTS_GOV_API_KEY),
pageSize: 25,
});
// Use the compiled schemas to transform native data into CommonGrants shape
const { toCommon } = grantsGovPlugin.schemas.Opportunity;
const opportunity = toCommon(rawGrantsGovData);
// Inspect what the plugin declares about itself
console.log(grantsGovPlugin.meta.sourceSystem); // "grants.gov"
console.log(grantsGovPlugin.meta.capabilities); // ["customFields", "transforms", "client"]
  • Positive consequences
    • Client stays singular — getClient() is memoized by definePlugin(), so one source system always produces one Client instance regardless of how many times getClient() is called
    • Top-level surface (meta, client, schemas, extensions) is short, closed, and stable — adding protocol objects adds a key under schemas only
    • Dependency injection works along functional lines: pass getClient, pass Schemas, pass Extensions as coherent units without needing to reassemble from per-object branches
    • mergeExtensions() / merge_extensions() operates on flat, serializable data at the root, not on deeply nested per-object branches
    • Per-object grouping inside schemas preserves the real coupling between native schema, CommonGrants schema, and bidirectional transforms — they share type signatures and change together
    • Mirrors the SDK module structure (client, schemas, extensions), so Plugin reads as a system-specific version of the existing SDK rather than a different mental model
    • toCommon/fromCommon can be plain hand-written functions or generated via buildTransforms() — plugin authors are not required to use a declarative mapping format
    • buildTransforms() accepts separate toCommonMapping and fromCommonMapping objects, reflecting that the two directions of a bidirectional transform are distinct
    • customFields is optional — the customFields-only config structure remains valid; existing plugin packages require only minimal code changes to adopt definePlugin()
    • All top-level Plugin fields are optional — adopters can start with only what they need and expand incrementally
  • Negative consequences
    • extensions (serializable config) and plugin (runtime object including client) are distinct concepts that adopters must understand separately
  • Backward compatible: Existing custom-fields-only plugins remain valid without changes
  • SDK-friendly: Config shape maps naturally to Pydantic/Zod one-model-at-a-time usage inside schemas
  • Language-agnostic config: The extensions JSON document uses camelCase keys (customFields, fieldType, sourceSystem) in both SDKs — Python source uses snake_case attributes with camelCase alias fields, matching the existing SDK convention
  • Clear naming: A single term — “plugin” — is used consistently across the registry catalog and SDK
  • Supports both capabilities: Custom field declarations and bidirectional transforms can coexist or be used independently; transforms may be hand-written or generated from declarative mappings
  • Incremental adoption: All top-level fields are optional, so adopters can start with only what they need
  • DI-friendly: Functional top-level keys can be passed as coherent units without reassembly
  • Stable surface: New protocol objects do not expand the top-level key set
  • Object-first structure with adapted model/schema (no separate Plugin class)
  • Pure object-first structure with “Plugin” for both registry and SDK
  • Functional top-level with per-object schema grouping (chosen)
CriteriaObject-first / Adapted SchemaPure object-first / PluginFunctional top-level / per-object schemas
Backward compatible
SDK-friendly
Language-agnostic config
Clear naming🟡
Supports both capabilities
DI-friendly🟡🔴
Stable surface🟡🔴

Option 1: Object-first structure, adapted model/schema (no separate Plugin class)

Section titled “Option 1: Object-first structure, adapted model/schema (no separate Plugin class)”

Instead of constructing a separate Plugin object, the SDK returns an extended version of the model/schema itself with the transform baked in. Adopters call native parse/validate methods directly on the returned object.

// TypeScript: createPlugin returns an extended Zod schema (ZodEffects), not a Plugin object
const opportunityPlugin = createPlugin(opportunitySchema, pluginConfig);
const opportunity = opportunityPlugin.parse(grantsGovData); // native Zod
const result = opportunityPlugin.safeParse(grantsGovData); // native Zod non-throwing
  • Pros
    • Very idiomatic — .parse() / safeParse() in Zod and .model_validate() in Pydantic are the expected call sites
    • No new runtime class name to explain; the adapted schema is still recognizably a schema
    • Backward compatible — both keys optional, custom_fields-only is valid
  • Cons
    • No named Plugin type to import, document, or type-hint against
    • Client, auth, and transport have no natural home in this model
    • DI requires passing each model’s plugin separately rather than a unified Schemas object — callers must accept one plugin per object rather than a single Schemas unit (DI-friendly: 🟡)
    • Top-level surface tracks the object list indirectly via function calls (createPlugin(opportunitySchema, ...), createPlugin(applicationSchema, ...)), but there is no stable type that enumerates supported objects (Stable surface: 🟡)
    • In Python, create_plugin must dynamically generate a new model class, which is less transparent

Option 2: Pure object-first structure, “Plugin” for both registry and SDK

Section titled “Option 2: Pure object-first structure, “Plugin” for both registry and SDK”

Config and runtime object both keyed by CommonGrants model name at the root. meta and client sit alongside object keys but are not themselves objects.

interface Plugin {
meta?: PluginMeta;
client?: Client;
Opportunity?: ObjectPluginConfig;
Application?: ObjectPluginConfig;
// ... one key per protocol object
}
  • Pros
    • Co-location: all of an object’s config is in one branch during authoring
    • Single unified term — Plugin is used for both the registry catalog and the SDK runtime object
  • Cons
    • Top-level keys track the protocol’s object list, which is long and open-ended — surface grows as protocol grows and raises questions about which of 100+ schemas belongs at the top level
    • Client, auth, and transport are system-level but must either be duplicated per object or kept implicit alongside object keys, creating an awkward mix of concerns
    • Filters attach to resource methods rather than schemas, and resource methods aren’t consistent across objects (e.g. opportunities.list/get/search vs applications.start/submit), creating a poor fit
    • DI requires reassembling a flat view across all object branches (e.g. an allSchemas helper) — working against the grain of the structure
    • mergeExtensions() must deeply merge nested per-object branches rather than operating on a flat serializable root

Option 3: Functional top-level with per-object schema grouping (chosen)

Section titled “Option 3: Functional top-level with per-object schema grouping (chosen)”

Top-level keys are functional (meta, client, schemas, extensions). Per-object grouping is used only inside schemas, where it reflects real coupling between native schemas, CommonGrants schemas, and bidirectional transforms.

interface Plugin {
meta?: PluginMeta;
client?: Client;
extensions?: PluginExtensions;
schemas?: Partial<Record<ExtensibleSchemaName, ObjectSchemas>>; // per-object grouping only here
}
  • Pros
    • Short, stable top-level surface — meta, client, schemas, extensions tracks a closed list regardless of how many protocol objects exist
    • Client is singular — getClient() is memoized by definePlugin(), so one source system always produces one Client instance
    • DI works along functional lines: pass getClient, pass Schemas, pass Extensions as units
    • mergeExtensions() operates on flat, serializable data at the root, not deeply nested per-object branches
    • Per-object grouping inside schemas preserves real coupling — native schema, CommonGrants schema, toCommon, and fromCommon share type signatures and change together
    • Mirrors the SDK module structure (client, schemas, extensions) — Plugin is a system-specific version of the existing SDK, not a different mental model
  • Cons
    • extensions (serializable config) and plugin (runtime object including client) are distinct concepts that adopters must learn separately