Before you begin: scope and permissions
- Confirm repository read access and organization policies for the approved export identity.
- Review repository, dependency, contributor and customer-code rights separately; public visibility is not a licensing determination.
- Approve the intended scope and use with the business owner; export access alone does not establish licensing rights.
- Use an approved encrypted working environment with restricted access and a retention deadline.
Separate source code from development conversations
GitHub hosts repository history and collaboration records such as issues, pull requests, reviews and discussions. Organizations may archive these for continuity or migration. A dataset connecting a defect to its fix needs the issue, relevant review and actual commit relationship; a ZIP of the default branch cannot provide that timeline.
Export repository files and history
For an authorized repository, GitHub’s backup guide describes a mirror clone:
git clone --mirror https://github.com/EXAMPLE/REPOSITORY.gitReplace the placeholders and use your approved authentication method without embedding credentials in a command or URL. A mirror contains Git history and references, not the site’s issue database. If LFS is used, collect its objects separately with Git LFS. A browser source ZIP is suitable for a code snapshot; it is not a history export.
Collect metadata and conversations separately
GitHub documents migration archives for selected repository metadata. The backup reference warns that these omit content including discussions, packages and LFS objects. Treat them as scoped archives, not a complete restorable backup.
For a dataset needing issue comments, review threads or discussions, design an approved API collector for those exact entities. GitHub documents a Discussions GraphQL API for discussion content; treat it as a distinct collection from issues and pull requests. Confirm the current endpoint permissions and pagination before implementation. Do not assume a migration archive or Git clone includes them. Save a collection matrix naming each entity type, method, unique count and exclusion.
Connect changes to their evidence
Keep repository identity, commit SHA, issue or pull request number, comment ID and timestamps. Numbers need repository context: issue 18 in one repository is unrelated to issue 18 elsewhere. Retain links between a pull request, its base and head, merge status, review observations and changed files when actually collected.
A synthetic training candidate might pair issue repo-a#18, review comment rc-22 and fixing commit abc123…. Preserve the reviewed revision, not merely today’s file content. An unmerged proposal should not be labeled as an accepted fix. Reverted commits and reopened issues deserve explicit outcome flags.
Normalize code without flattening history
Identify generated files, dependencies, vendored directories and copied examples before deduplication. A generated bundle can dominate byte counts while adding little independent engineering evidence. Keep file language and path context where permitted, but check whether paths expose customer or employee identities.
Split by repository or related change family when a task needs generalization. A patch in training and the same patched file in evaluation can leak the answer. Preserve license notices and track which candidate examples depend on excluded third-party material.
Review rights and sensitive information
An organization may control a private repository while still holding contractor contributions, licensed dependencies or customer code under restrictions. Separate authority to export from authority to license source and discussion content. Scan all included history for credentials and secrets; deleting a credential from the latest file does not remove it from prior commits. Keep only a reviewed derivative for external evaluation.
Troubleshoot and assess buyer fit
Why are issues missing from the clone? They are service metadata, not Git objects. Use a separately verified collection method.
Why are large files only pointers? Check the LFS inventory and whether actual objects were collected.
Potential tasks include code-review evaluation or defect-resolution reasoning. Use complete, rights-cleared examples and distinguish automated labels from verified outcomes. Report language mix, test evidence, history coverage and missing collaboration records before discussing any sample or commercial terms.
Release checklist
- Inventory Git objects, LFS assets, issues, reviews and discussions as separate coverage categories.
- Scan the full included commit history for secrets, not just the current branch.
- Reconcile source counts and date coverage; record excluded or unavailable records.
- Validate identifiers, relationships, timestamps and field types after transformation.
- Document provenance, privacy review, third-party restrictions and the authorized release approver.
- Scan the release files for personal information, secrets and confidential material; test a reviewed sample before transfer.
Have a proprietary dataset?
Find out whether it may be suitable for AI licensing. Start with a high-level description of the source, date range, volume, rights status, and preparation work still needed.
Assess dataset readinessUse the existing buyer application when ready. Share only authorized information. Do not submit raw confidential records, credentials, or personal data in the initial assessment.
Sources & verification
Official documentation reviewed 2026-10-15. Procedures and access rules can change. These are documentation-based instructions, not a live export from your account. Verify your product version, permissions and retention settings before running a job.