How to Prepare Enterprise Data for AI Training

Start with a task and an allowed scope

An enterprise export is raw material, not a finished training dataset. Preparing it means selecting relevant observations, preserving their context, controlling sensitive information and documenting limitations. A support-answer task needs different evidence from opportunity forecasting or document retrieval. Write a one-page specification naming the proposed task, allowed fields, unit of observation, historical cutoff and prohibited uses before collecting.

This workflow is an editorial preparation method, not a universal model-provider format or a determination of licensing rights. The Datasheets for Datasets paper motivates documenting dataset purpose, composition and use. Use that principle to make your own scope and limitations reviewable.

Inventory systems before expanding collection

List source owners, record types, retained dates, stable IDs, expected relationships and known export gaps. Use a small pilot from the appropriate source guide: Slack threads, Salesforce objects or Zendesk conversations. Do not assume a dashboard count measures complete conversations or independent examples.

For each source, record a collection method and reconcile the output with a known sample. Save query parameters, export options, completion time and errors without including credentials. Keep a restricted raw copy and a separate working copy so mistakes are reversible. Avoid copying production data into public tools for convenience.

Transform in explicit stages

Use a reproducible pipeline with recorded inputs and outputs:

1. Parse each source format and log malformed records.
2. Normalize timestamps, encoding and field names while retaining source values.
3. Rebuild relationships and mark missing parents or assets.
4. Identify duplicates at the correct business grain.
5. Remove or transform sensitive content using an approved policy.
6. Assemble task examples and evaluate them against acceptance criteria.

Keep a count ledger: collected, excluded, transformed, rejected and released. If 10,000 collected tickets become 6,400 complete examples, explain the other 3,600 by reason. Do not silently “fix” missing values with plausible text.

Preserve context and avoid outcome leakage

Synthetic example: a support conversation has a failed answer, a corrected instruction and a confirmation. Keep the sequence and differentiate the unsuccessful answer from the target. For a pre-resolution prediction task, exclude fields recorded after the decision cutoff from the input. The same record may require different preparation for different tasks.

Group train and evaluation partitions by conversation, customer incident, document family or repository where appropriate. Random message-level splitting can put the answer to an evaluation question in the training set. Document grouping decisions and residual overlap rather than claiming a perfectly independent split without checking.

Review rights and release evidence

Source administration, export permission, privacy compliance and authority to license are separate review questions. Record customer, employee, contractor and third-party contributions and unresolved restrictions. Redaction does not erase intellectual-property or confidentiality obligations. Stop external transfer for unresolved scope even if the technical pipeline passes.

Use provenance records to connect derivatives to source jobs; the W3C primer supplies a general model for entities, activities and responsible agents. Maintain a manifest describing the actual release, files, checks and limitations.

Resolve common readiness questions

Should we collect everything first? Start with the smallest permitted pilot that tests usefulness and missing context. Expand after reviewing its costs and restrictions.

Does valid JSONL mean the dataset is training-ready? No. Syntax, task fit, quality and rights each need their own checks.

What should we describe to an evaluator? Provide domain, language, coverage, unique observation counts, context completeness and unresolved review work. Potential buyer fit depends on these and the buyer’s task; preparation does not guarantee demand or price.