7.15. Input validation
Up: 7. Developer guide
Prev: 7.14. Authorization security
Next: 7.16. Dependency updates
Sections:
- Overview
- Defense in depth
- Form validation with Pydantic
- CSRF protection
- Validation rules by input type
- Data integrity validation
- Business logic validation
- Output encoding
- File upload security
- Injection prevention
- Implementation references
Overview
Input validation is critical for ATR's security posture. As a system that handles cryptographic signatures and release artifacts, ATR must ensure that all user input is properly validated before processing. This page documents the validation strategies and patterns used throughout the codebase.
Defense in depth
ATR employs multiple layers of validation:
- Transport layer: HTTPS required, enforced by httpd
- Request layer: ATR enforces the request body limit set by
MAX_CONTENT_LENGTH - Form layer: Pydantic models validate structure and types
- Application layer: Business logic validation in route handlers, interaction helpers, and storage writers
- Database layer: SQLAlchemy ORM with parameterized queries, plus constraints
- Markdown layer: Via
cmarkgfm - Output layer: Jinja2 auto-escaping for HTML output
Each layer provides independent protection, so a failure in one layer does not compromise the system.
Form validation with Pydantic
All form inputs in ATR are validated through Pydantic models
defined in form.py. The base class for forms is Form
(Form), which extends Pydantic's BaseModel.
Defining form fields
Form fields are defined using Python type annotations and the label (label)
function:
class ExampleForm(Form):
name: str = label("Project name", "Enter the project name")
count: int = label("Count", widget=Widget.NUMBER)
email: EmailStr = label("Contact email", widget=Widget.EMAIL)
The label function accepts a description (shown to users), optional documentation, and an optional
widget hint for rendering.
Validation process
When a form is submitted, ATR:
- Extracts form data from the request via
quart_request(quart_request) - Passes the data to the Pydantic model for validation
- If validation fails, collects errors via
flash_error_data(flash_error_data) - Displays errors to the user with
flash_error_summary(flash_error_summary) - If validation succeeds, proceeds with the validated data
Pydantic provides built-in validators for common types (strings, integers, emails, URLs) and supports custom validators via decorators.
Custom validators
For complex validation logic, use Pydantic's @model_validator decorator:
from pydantic import model_validator
class ReleaseForm(Form):
version: str = label("Version")
@model_validator(mode="after")
def validate_version_format(self):
if not re.match(r"^\d+\.\d+\.\d+", self.version):
raise ValueError("Version must start with X.Y.Z")
return self
CSRF protection
All POST forms must include a CSRF token. The token is generated by csrf_input
(csrf_input) and validated automatically by Quart-WTF:
def csrf_input() -> htm.VoidElement:
csrf_token = utils.generate_csrf()
return htpy.input(type="hidden", name="csrf_token", value=csrf_token)
In templates, include the CSRF token in every form:
<form method="post">
{{ csrf_input() }}
<!-- other form fields -->
</form>
The CSRF token is tied to the user's session and validated on form submission. Requests without a valid CSRF token are rejected. When using the form module renderer, the CSRF token is added automatically.
Validation rules by input type
ASF User IDs
User IDs are validated against a strict pattern in principal.py:
if not re.match(r"^[-_a-z0-9]+$", user):
raise CommitterError("Invalid characters in User ID")
Only lowercase alphanumeric characters, hyphens, and underscores are permitted.
Email addresses
Email validation uses Pydantic's EmailStr type, which implements RFC 5322 validation:
from pydantic import EmailStr
class ContactForm(Form):
email: EmailStr = label("Email address")
URLs
URL validation uses Pydantic's HttpUrl type:
from pydantic import HttpUrl
class LinkForm(Form):
website: HttpUrl = label("Website URL")
Version strings
Version strings are validated according to project-specific patterns. The general pattern allows semantic versioning with optional suffixes:
VERSION_PATTERN = re.compile(r"^[0-9]+\.[0-9]+.*$")
Committee and project names
Committee and project names are validated against the set of known committees and projects from LDAP and the ASF project database. Unknown names are rejected.
File names
Release upload paths use safe.RelPath, which rejects absolute paths,
directory traversal, null bytes and most dotfiles. These rules apply to every upload method,
including rsync and SVN imports. For file content checks, see File handling.
Data integrity validation
Beyond input validation, ATR performs data integrity validation on database records using
validate.py. This catches inconsistencies that may have been introduced by
bugs, migrations, or manual database edits.
Committee validation
Committee records are not created through a form. They are synced from LDAP and Whimsy by
_update_committees (_update_committees), so their fields have no
Pydantic input layer, and the data integrity validators are the main place each field is confirmed.
The committee (committee) function checks:
keymust be a valid committee key (thesafe.CommitteeKey(CommitteeKey) character set)namemust be set, trimmed, and not prefixed with "Apache "committee_members,committers, andrelease_managersentries must each look like an ASF UIDchild_committeesmust be empty (not used)
Per-field coverage:
| Field | Input layer | Data integrity |
|---|---|---|
key |
none (LDAP project name) | valid committee key |
name |
none (Whimsy) | set, trimmed, no "Apache " prefix |
is_podling |
set during sync | type-enforced (bool) |
parent_committee_key |
none | foreign key |
committee_members |
none (LDAP) | ASF UID format |
committers |
none (LDAP) | ASF UID format |
release_managers |
none | ASF UID format |
child_committees |
none | must be empty |
Project validation
Projects do have an input layer. AddProjectForm (AddProjectForm)
validates the key and display name at creation, and
EditVersionSchemeForm (EditVersionSchemeForm) rejects a
version_pattern or cycle_match that is not a valid regex (and a cycle_match with no capture
group). Calver projects describe their cycles with a calver_format date format instead of a regex;
see Calver cycle formats below. The data integrity validators mirror these
and cover fields that have no form.
The project (project) function checks:
keymust be a valid project key (thesafe.ProjectKey(ProjectKey) character set)categorymust use comma-separated labels without colonscommittee_keymust be set (project must be linked to a committee)createdtimestamp must be in the pastcreated_by, if set, must look like an ASF UIDcycle_match, if set, must be a regex with at least one capture groupcalver_format, if set, must be a valid date format with at least one calendar fieldnamemust be set and start with "Apache "programming_languagesmust use comma-separated labels without colonsversion_pattern, if set, must be a compilable regex
Per-field coverage:
| Field | Input layer | Data integrity |
|---|---|---|
key |
safe.ProjectKey (charset, lowercase) + AddProjectForm (committee-prefixed) |
valid project key |
name |
AddProjectForm and EditMetadataForm (Apache prefix, case rules); configuration API currently accepts names without the prefix |
set, "Apache " prefix |
status |
none (enum) | type-enforced (enum) |
description |
EditMetadataForm (free text) |
none |
category |
AddCategoryForm (free text) |
comma-separated, no colons |
programming_languages |
AddLanguageForm (free text) |
comma-separated, no colons |
version_method |
EditVersionSchemeForm (enum) |
type-enforced (enum) |
version_pattern |
EditVersionSchemeForm (regex validity) |
compilable regex |
cycle_match |
EditVersionSchemeForm (regex and capture group) |
regex with capture group |
calver_format |
EditVersionSchemeForm (date format) |
valid format with a calendar field |
branch_template |
EditVersionSchemeForm (free text) |
none, not currently enforced |
committee_key |
safe.CommitteeKey |
must be set |
created |
set at creation | in the past |
created_by |
none | ASF UID format |
Calver cycle formats
Calver projects declare the shape of their version strings with a date format string rather than a regex. The format is a sequence of tokens with literal separators between them:
| Token | Matches | Role |
|---|---|---|
YYYY / YY |
four- or two-digit year | ordering |
MM / M |
zero-padded or 1-2 digit month | ordering |
DD / D |
zero-padded or 1-2 digit day | ordering |
N |
a numeric serial (a patch ordinal within a calendar line) | ordering |
Wrap the part of the version that names the cycle in parentheses. A format with no parentheses keeps
a single release track; one with a parenthesised span splits releases into one cycle per matched
value. Trailing tokens are optional, so one format covers mixed-granularity histories - YYYYMMDD
matches both 20130710 and 201407, reading the missing day as zero.
Examples:
| Version | Format | Cycle | Notes |
|---|---|---|---|
2020.6.24 |
YYYY.M.D |
default | non-padded month and day |
2025-11-03 |
YYYY-MM-DD |
default | dashed date |
20130710 |
YYYYMMDD |
default | run-together, day optional |
09.04.01 |
(YY.MM).N |
09.04 |
the YY.MM line is the cycle, N the patch |
Release validation
The release (release) function checks:
createdtimestamp must be in the pastnamemust match the expected pattern for project and version- A draft's latest revision directory must exist, and a candidate's or preview's must not be empty
- A released version must have no unfinished directory, except while its finalisation task is queued or active
package_managersmust be empty (not used)releasedtimestamp must be in the past or Nonesbomsmust be empty (not used)- Vote logic must be consistent (cannot have
vote_resolvedwithoutvote_started)
Stored consistency checks
These checks find stored records that disagree with each other, and stored settings that ATR cannot use. Some write paths do not enforce these rules, as noted below. The checks report problems without blocking any action.
| Rule | Validator | Source |
|---|---|---|
| A draft has no vote mode, resolution time, podling thread, voted revision, or vote thread URL. A failed or cancelled vote may leave its start time and vote serial number behind. | release_vote_fields |
_resolve_transition |
| A candidate has a vote start time and mode, but no resolution time or vote thread URL. A preview has a start time and a mode, and also a resolution time if it has a vote serial number. A release catalogued from the distribution area may have no vote fields at all. | release_vote_fields |
start_vote_no_commit, _resolve_transition |
| A vote serial number requires a vote start time. A voted revision requires a vote serial number, a start time, and a mode. Releases that voted before ATR recorded vote serial numbers or voted revisions may lack them. | release_vote_prerequisites |
start_vote_no_commit |
| A candidate's or preview's voted revision, when recorded, is still its latest revision. | release_voted_revision |
start_vote_no_commit |
| A draft, candidate, or preview belongs to an active project. Admins can retire a project that has such releases, which leaves them unable to progress. | release_project_active |
ensure_project_active |
| An archive date or archive source requires the release to be archived. Older archived releases may have neither. This check is temporarily disabled. | release_archived |
archive_release_core |
| A release's cycle is the one its project's current cycle matcher gives for its version, or the default cycle if matching fails. | release_cycle |
reassign_release_cycles |
| A calver format is set only on calver projects, is valid, and generates the project's stored cycle matcher. | project_calver_format |
edit_version_scheme, _apply_version_scheme |
| Release and policy download path suffixes are valid paths after filling in their template variables. An unset release suffix uses the policy suffix, and an empty one means the distribution root. Both are valid. | release_download_path_suffix, project_download_path_suffix |
validate_download_path_suffix |
Recipient defaults have only vote and announce entries. Each has a to string and cc and bcc lists of strings, and every address is on apache.org or one of its subdomains. The web forms and policy API do not check these addresses when saving them. An entry with an empty to and empty lists is allowed. |
project_recipient_defaults |
_set_recipient_defaults, _validate_recipient |
The minimum voting period is unset, 0, or 72 to 168 hours. Template and RAT excludes URLs use HTTPS on an allowed host. A security contact, if set, is security@apache.org or the committee's own security@ address. |
project_min_hours, project_fetch_urls, project_security_contact |
validation.py |
A Trusted Publishing repository name, if set, has no slash, and every workflow path starts with .github/workflows/. |
project_trusted_publishing |
validate_trusted_publishing_constraints |
| Each release finalisation task has a completed SVN publication task for the same project, version, and revision. That task has a valid stored result that records the SVN revision. This only audits the task records. Finalisation itself verifies the published files. | release_finalise_publication |
release |
| A pending quarantine has a queued or active validation task. Without one, the upload stays pending and blocks the vote. | quarantine_pending |
create_revision_with_quarantine |
| A staging or pending quarantine has no completion time and belongs to a draft. A failed or acknowledged quarantine has a completion time. A recorded prior revision belongs to the same release, even if that revision no longer exists. | quarantine_completed, quarantine_release, quarantine_revision in quarantines |
create_revision_with_quarantine, quarantine.py |
| Each release's revision counter is at least its highest revision sequence number. For a draft, candidate, or preview, the vote counter is at least the highest vote serial number on the release, its ballots, or its vote initiation tasks. Gaps are allowed, and so are counters that outlive their release. | allocation_counters |
populate_revision_sequence_and_key, start_vote_no_commit |
| An artifact's release key, if set, matches its project and version. A recorded signer or signature digest requires a signature path, and a digest is 64 lowercase hexadecimal digits. | artifact_release_key, artifact_signature in artifacts |
release, catalogue_release |
| A system token has no owner, and an ordinary token's owner is its creator. | personal_access_token_owner in personal_access_tokens |
tokens.py |
| Check result ignore patterns compile. | check_result_ignores |
validate_ignore_pattern |
Some rules depend on who is acting or on the moment of the request, so stored records cannot show whether they held. ATR enforces these only when the action happens:
- Starting a vote or announcing a release proceeds only if the phase and latest revision still match the request. Resolving a vote checks the phase and vote serial number in the same way. Publishing rechecks the latest revision under a write lock. A stale request cannot move a release on.
- A vote can start only when no checks are queued or running, no upload is waiting in quarantine, there are no blockers, the draft has at least one file, and the required concern groups are acknowledged. See Vote initiation.
- The permitted vote and announce recipients depend on the acting user, the committee and its mailing lists, the project's stored defaults, the
ATR_STATUSsetting, and the podling vote round. Seepermitted_voting_recipientsandpermitted_announce_recipientsinutil.py. - An announcement requires a completed SVN publication of the exact preview revision, a recorded distribution for every file tag mapping, and an unchanged subject template. ATR also checks that the published files are available from every download server. If a server cannot be checked, the release manager can acknowledge this and continue, but missing or blocked files always stop the announcement. See the announcement writer.
- Only the
.asf.yamlroute requires vote recipients on the committee's ownapache.orgsubdomain. SeeProjectConfigArgs. - The
.asf.yamlroute refuses a cycle matcher for a calver project, because ATR generates it from the calver format. See_apply_version_scheme. - A release version must match the project's version pattern when the release starts. Later pattern changes do not affect existing releases. See
_validate_version.
Running validation
Data integrity validation can be run via the admin interface or programmatically:
async for divergence in validate.everything(data):
print(f"{divergence.source}: {divergence.divergence}")
These validators are complementary to the live checks described below. Data integrity validation inspects stored records for drift or corruption. Business logic validation stops inconsistent actions before ATR accepts them.
Business logic validation
Field validation is only the first step. ATR also checks whether an action still makes sense in the
wider state of the release. These rules compare data across releases, revisions, committees, queued
tasks, stored policy, and message delivery settings. They live mainly in
interaction.py, storage writers, mail.py, and
shared helpers in util.py.
Vote initiation
Before a vote can start, release_ready_to_start_vote
(release_ready_to_start_vote) checks that the release still has a latest revision and that the
release is still attached to a committee.
That same validation step then checks the surrounding release state. The user must be a release
manager for the project, meaning a PMC member or a designated committer release manager, or an ATR
administrator. The latest revision must have no blocker results, and the release candidate draft
must contain files. When ATR actually promotes the release into the voting phase,
start_vote_no_commit (start_vote_no_commit) adds a task
state check and refuses the transition while queued or active tasks still exist for that revision.
The same writer atomically enforces that the transition can only happen from the expected latest revision, using a caller supplied expected revision and a latest revision subquery in the UPDATE statement as a compare-and-swap guard. This binds vote initiation to release phase, revision state, policy, committee membership, check results, file storage, and task execution rather than to form fields alone.
Trusted Publishing
Trusted Publishing settings are validated when they are stored and again when they are used. On
write, validate_trusted_publishing_constraints
(validate_trusted_publishing_constraints) and policy.py
normalize the configured repository name, branch, and workflow paths. The repository name is
optional and is stored without a slash. Every workflow path must begin with .github/workflows/.
At request time, _trusted_project_checks (_trusted_project_checks)
and _trusted_project (_trusted_project) compare the GitHub token
claims with the stored policy. The repository must be under apache. The workflow reference must
begin with that same repository, must include a git ref, and must resolve to a workflow path under
.github/workflows/. ATR then looks up the project by the phase specific workflow path that was
stored for compose, vote, or finish, and by repository: the one named in the policy when it sets
one, or otherwise any GitHub repository in the project metadata. Distribution callbacks add one more
contextual check in trusted_jwt_for_dist (trusted_jwt_for_dist),
which refuses the request unless the named release exists and is in the expected phase for the
requested operation. The cryptographic validation of the token itself is described in
authentication security.
Email delivery
Email validation in ATR also depends on context. validate_email_recipients
(validate_email_recipients) requires a primary recipient and rejects duplicate addresses across
To, Cc, and Bcc. send (send) then requires the sender to use
@apache.org, and _validate_recipient (_validate_recipient) rejects any
envelope recipient outside @apache.org and its subdomains. This means that vote and release mail
must go to ASF controlled addresses even if the address itself would be syntactically valid.
Output encoding
ATR uses Jinja2 for templating with auto-escaping enabled by default. All variables rendered in templates are automatically HTML-escaped:
<!-- This is safe; user_input is escaped -->
<p>Hello, {{ user_input }}</p>
When HTML output is intentionally generated (e.g., via htpy), it must be explicitly marked safe
using markupsafe.Markup:
import markupsafe
safe_html = markupsafe.Markup("<strong>Bold</strong>")
For Markdown rendering, ATR uses
markupsafe.Markup(cmarkgfm.github_flavored_markdown_to_html(markdown_text)), which safely filters
dangerous input before rendering.
Never mark user-controlled data as safe without proper sanitization.
File upload security
See File handling for upload limits, format checks, archive quarantine and downloads.
Injection prevention
SQL injection
ATR uses SQLAlchemy ORM exclusively for database access. All queries use parameterized statements:
# Safe: parameterized query
result = await session.exec(
select(Project).where(Project.key == project_name)
)
Direct SQL string concatenation is never used.
Cross-site scripting (XSS)
XSS is prevented through:
- Jinja2 auto-escaping (enabled by default)
markupsafe.Markupfor trusted HTML only- Content Security Policy headers set by ATR in server.py for application responses
Path traversal
Path traversal is prevented by:
- Using
pathlib.Pathfor all file operations - Validating that paths remain within expected directories
- Rejecting file names containing path separators
For form fields that accept file or directory paths, always use form.RelPath (or
form.RelPathList for multiple paths). These types automatically call
to_relpath() (to_relpath), which rejects path traversal sequences, absolute
paths, and empty values at the Pydantic validation layer. This is the preferred approach because it
prevents path traversal before the handler code runs.
For cases outside of form validation (e.g., URL route parameters), use
form.to_relpath() (to_relpath) directly, or validate manually:
import pathlib
base = pathlib.Path("/allowed/directory")
user_path = base / user_filename
# Verify the resolved path is still under base
if not user_path.resolve().is_relative_to(base.resolve()):
raise ValueError("Path traversal detected")
Command injection
ATR safeguards against command injection as much as possible. Where external commands are necessary (e.g., GPG operations), arguments are passed as lists, never as shell strings:
import subprocess
# Safe: arguments as list
subprocess.run(["gpg", "--verify", signature_file, data_file])
# Unsafe: never do this
subprocess.run(f"gpg --verify {signature_file} {data_file}", shell=True)
Implementation references
form.py- Form definitions, validation, and renderingdb/interaction.py- Contextual validation for vote initiation and Trusted Publishingstorage/writers/policy.py- Trusted Publishing policy validation and normalisationstorage/writers/release.py- Release phase transition checks and task state guardsvalidate.py- Data integrity validatorsutil.py- Shared validation helpers including path handling, recipients, and Trusted Publishingmail.py- Sender and recipient domain enforcementhtm.py- HTML generation utilities