How to Create a Dataset Manifest for AI Licensing

Make the release identifiable

A dataset manifest is a machine-readable inventory of one specific package. It helps a reviewer know which files exist, what they mean and whether they match the reviewed version. It does not grant a license or prove privacy clearance. Keep it separate from the restricted provenance register and from the executed agreement.

The W3C DCAT vocabulary provides concepts for cataloging datasets and distributions. The compact JSON example below is a proposed internal schema, not a DCAT conformance claim or a required format for AI buyers. Agree a recipient format before delivery if one is specified.

Define the minimum useful fields

Include dataset ID, release version, creation time, description, observation unit, source categories, covered dates, languages, schema reference, file paths, media types, byte counts, record counts and checksums. Add quality-summary and rights-review references, known exclusions, contact role and applicable use restrictions or agreement reference.

Distinguish released examples from raw collected records. For PostgreSQL, include table/view schema and null rules. For Workspace, distinguish original documents from extracted text. Every path should be relative to the package root and refer to an actual approved file.

Use an explicit synthetic example

{
  "dataset_id": "support-workflows-synthetic",
  "release_version": "1.0.0",
  "created_at": "2026-10-15T00:00:00Z",
  "observation_unit": "conversation",
  "schema": "schemas/conversations-v1.json",
  "files": [{
    "path": "data/conversations.jsonl",
    "media_type": "application/x-ndjson",
    "record_count": 2,
    "bytes": 420,
    "sha256": "REPLACE_WITH_COMPUTED_64_HEX_DIGEST"
  }],
  "known_gaps": ["Attachments excluded from this synthetic example"],
  "rights_review": {"status": "example_only", "evidence_ref": null},
  "license_agreement_ref": null
}

The counts and paths illustrate a schema and do not identify a real package. Replace them with computed values. Null agreement status means no agreement is asserted. Do not replace it with an invented license merely to complete the file.

Compute hashes from final bytes

For Python 3.11 or later, the hashlib reference documents file hashing:

import hashlib
from pathlib import Path

path = Path("release/data/conversations.jsonl")
with path.open("rb") as stream:
    digest = hashlib.file_digest(stream, "sha256").hexdigest()
print({"path": "data/conversations.jsonl", "bytes": path.stat().st_size,
       "sha256": digest})

Compute record counts by parsing the actual format, not by guessing from byte size. A checksum establishes byte identity when compared with a trusted manifest; it does not establish authenticity, rights, completeness or safety. Protect the manifest through the agreed transfer and verification process.

Validate and version the package

Verify every listed file exists, no unlisted sensitive files entered the archive, paths stay inside the release and the schema validates the data. Check unique record IDs, relationships and split groups. Recompute checksums after any final redaction, newline conversion or repackaging that changes file bytes.

Keep a release history and a clear replacement policy. If a correction changes the data, issue a new version and manifest rather than silently reusing the old version. Record what changed and how recipients should handle previous copies according to the agreement.

Document limitations and assessment status

Summarize missing dates, excluded assets, uncertain labels and quality findings with evidence references. Keep detailed contracts, identity maps and raw source locations in restricted storage. A recipient should understand the package without receiving secrets.

Can the manifest serve as a license? No. Link a reviewed agreement or clearly state that authorization is pending.

Should it list raw confidential sources? Only approved non-sensitive references belong in the release. Use internal evidence IDs for the rest.

A manifest makes a candidate concrete and reviewable. Buyer fit still requires task, quality, security and rights evaluation through the existing assessment process.