Skip to content

docs: Add mock server tests documentation for connectors #68667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: Add mock server tests documentation for connectors #68667

wants to merge 1 commit into from

Conversation

@devin-ai-integration
Copy link
Contributor

What

Adds comprehensive documentation for mock server tests in Airbyte connectors. This documentation describes the testing framework available in airbyte-python-cdk and provides best practices for writing effective mock server tests.

Requested by [email protected] ([email protected]) via Slack.

Link to Devin run: https://app.devin.ai/sessions/4db973c08aa34c24afd09a5efa20aa7f

How

Created a new documentation file at docs/platform/connector-development/connector-mock-server-tests.md following the format of connector-breaking-changes.md. The documentation includes:

  1. Framework Overview: Core components (HttpMocker, HttpRequest, HttpResponse, Response Builders) and test utilities (EntrypointWrapper, CatalogBuilder, StateBuilder)
  2. Test Organization: Recommended directory structure for mock server tests
  3. Best Practices: Four key principles with good/bad examples:
    • Keep tests minimal (don't configure unnecessary values)
    • Make tests expressive (show values that affect test outcomes)
    • Ensure tests are fast (disable sleep/delays with monkey patching)
    • Avoid fragile tests (don't fail on new API fields)
  4. Additional Best Practices: Builder pattern usage, response templates, common test scenarios
  5. Complete Example: Full working example demonstrating all concepts
  6. Reference Examples: Links to source-stripe and source-amazon-seller-partner as comprehensive examples

Review Guide

  1. Documentation Build: Verify the documentation builds correctly in Docusaurus/Vercel preview
  2. Code Examples: Check Python code examples for syntax/API correctness (especially imports and method signatures)
  3. Links: Verify all links work:
    • GitHub links to airbyte-python-cdk
    • Links to source-stripe and source-amazon-seller-partner examples
    • Internal doc links
  4. Best Practices Accuracy: Confirm the best practices align with team standards and actual framework usage
  5. Formatting: Check markdown formatting, code block syntax highlighting, and consistency with other docs

User Impact

Connector developers will have comprehensive documentation on how to write effective mock server tests. This will:

  • Help developers write faster, more reliable tests
  • Enable AI systems (like Devin) to generate mock server tests following best practices
  • Reduce time spent debugging brittle or slow tests
  • Improve overall test quality across connectors

Note: I encountered pnpm install errors when attempting to verify the local documentation build. The documentation should be reviewed in the Vercel preview to ensure it renders correctly.

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

This is documentation-only with no code changes.

@devin-ai-integration
Copy link
Contributor Author

Original prompt from [email protected] 
Received message in Slack channel #ask-devin-ai:

@Devin I would like to create documentation regarding mock server tests similar to docs/platform/connector-development/connector-breaking-changes.md in the airbytehq/airbyte repository. This would describe the testing framework available <https://github.com/airbytehq/airbyte-python-cdk/tree/main/airbyte_cdk/test|here> and summary the best practices.

We have a couple of good examples of mock server tests like <https://github.com/airbytehq/airbyte/tree/d1add7b8b50e185efe209fe8d23c906d87c3734f/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/integration|source-amazon-seller-partner> and <https://github.com/airbytehq/airbyte/tree/d1add7b8b50e185efe209fe8d23c906d87c3734f/airbyte-integrations/connectors/source-stripe/unit_tests/integration|source-stripe>.

Here are a couple of qualities of good mock server tests:
• They are minimal i.e. they don't configure things that are not needed. For example, a test that validates that records are extracted normally do NOT configure values for the entry in the HTTP response from the API (except if the source is doing filtering in which case the fields should be made explicit)
• They are expressive i.e. they show things that affect the test. One example of this is that if the success of a test depend on a property value from the API, this should be provided in the context of the test and it shouldn't rely on the values defined in `resource/http/response/<response>.json`
• They are fast which means that if a test is sleeping, we should disable the sleeping using monkey patch
• They are not fragile: If a test fails because a new field is added by the API provider, than it is not a good test except if there is something important about this field (filtering, transformation, etc...)
The goal for this documentation is for devs to create more mock server tests but also for AI like you to generate some.
Thread URL: https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/... (45 chars truncated...)

@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
  • JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
    • /bump-bulk-cdk-version type=patch changelog='foo' - Bump the Bulk CDK's version. type can be major/minor/patch.
  • Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

github-actions[bot]
github-actions bot reviewed Oct 27, 2025
View reviewed changes
docs/platform/connector-development/connector-mock-server-tests.md

Organize mock server tests in a dedicated directory structure:

```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]

docs/platform/connector-development/connector-mock-server-tests.md
Tests should only configure values that are necessary for the specific scenario being tested. Avoid setting unnecessary fields or using complete API responses when a minimal response suffices.

**Good Example** (minimal):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

docs/platform/connector-development/connector-mock-server-tests.md
```

**Bad Example** (excessive configuration):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

docs/platform/connector-development/connector-mock-server-tests.md
Tests should explicitly show values that affect the test outcome. If a test depends on a specific field value, set it in the test context rather than relying on values in template files.

**Good Example** (expressive):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

docs/platform/connector-development/connector-mock-server-tests.md
```

**Bad Example** (implicit dependencies):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

docs/platform/connector-development/connector-mock-server-tests.md
- Lookback window behavior

**Error Handling Tests**:
- Rate limiting (429) with retry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "- Rate limiting (429) with ret..."]

docs/platform/connector-development/connector-mock-server-tests.md
- Client errors (400) with appropriate failure type

**Example Test Suite Structure**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

docs/platform/connector-development/connector-mock-server-tests.md
Here's a complete example demonstrating best practices:

**config.py**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

docs/platform/connector-development/connector-mock-server-tests.md
```

**request_builder.py**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

docs/platform/connector-development/connector-mock-server-tests.md
```

**test_events.py**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```python"]

github-actions[bot]
github-actions bot reviewed Oct 27, 2025
View reviewed changes
docs/platform/connector-development/connector-mock-server-tests.md
@@ -0,0 +1,646 @@
# Mock Server Tests for Connectors

Mock server tests provide a fast, reliable way to test connector behavior without making actual API calls. These tests use the Airbyte Python CDK's mock HTTP framework to simulate API responses and verify connector logic.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'CDK's'?

docs/platform/connector-development/connector-mock-server-tests.md

## Why Use Mock Server Tests

Mock server tests offer several advantages over integration tests:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[write-good.Weasel] 'several' is a weasel word!

docs/platform/connector-development/connector-mock-server-tests.md

### Core Components

**HttpMocker**: Context manager and decorator that intercepts HTTP requests and returns mocked responses.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Colons] ': C' should be in lowercase.

docs/platform/connector-development/connector-mock-server-tests.md
# Run connector code
```

**HttpRequest**: Represents an HTTP request with URL, query parameters, headers, and body.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Colons] ': R' should be in lowercase.

docs/platform/connector-development/connector-mock-server-tests.md
)
```

**HttpResponse**: Represents an HTTP response with body, status code, and headers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Colons] ': R' should be in lowercase.

docs/platform/connector-development/connector-mock-server-tests.md

### Test Utilities

**EntrypointWrapper**: Executes connector operations (read, check, discover) and captures output.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Colons] ': E' should be in lowercase.

docs/platform/connector-development/connector-mock-server-tests.md
assert len(output.records) == 10
```

**CatalogBuilder**: Constructs configured catalogs for testing.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Colons] ': C' should be in lowercase.

docs/platform/connector-development/connector-mock-server-tests.md
catalog = CatalogBuilder().with_stream("users", SyncMode.incremental).build()
```

**StateBuilder**: Constructs state messages for incremental sync testing.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.Colons] ': C' should be in lowercase.

docs/platform/connector-development/connector-mock-server-tests.md

### 4. Avoid Fragile Tests

Tests should not fail when the API provider adds new fields to responses. Only validate fields that are important for the connector's functionality (filtering, transformation, cursor tracking, etc.).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [vale] reported by reviewdog 🐶
[Google.WordList] Use 'capability' or 'feature' instead of 'functionality'.

docs/platform/connector-development/connector-mock-server-tests.md
- Subsequent sync with state
- State message generation
- Cursor field handling
- Lookback window behavior
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚫 [vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'Lookback'?

github-actions[bot]
github-actions bot reviewed Oct 27, 2025
View reviewed changes
docs/platform/connector-development/connector-mock-server-tests.md
Tests should only configure values that are necessary for the specific scenario being tested. Avoid setting unnecessary fields or using complete API responses when a minimal response suffices.

**Good Example** (minimal):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
```

**Bad Example** (excessive configuration):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
Tests should explicitly show values that affect the test outcome. If a test depends on a specific field value, set it in the test context rather than relying on values in template files.

**Good Example** (expressive):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
```

**Bad Example** (implicit dependencies):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
Mock server tests should execute quickly. Disable any sleep or delay mechanisms using monkey patching.

**Good Example** (fast):
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
- Lookback window behavior

**Error Handling Tests**:
- Rate limiting (429) with retry
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
- Rate limiting (429) with retry
- Rate limiting (429) with retry

docs/platform/connector-development/connector-mock-server-tests.md
- Client errors (400) with appropriate failure type

**Example Test Suite Structure**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
Here's a complete example demonstrating best practices:

**config.py**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
```

**request_builder.py**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

docs/platform/connector-development/connector-mock-server-tests.md
```

**test_events.py**:
```python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
```python
```python

@github-actions
Copy link
Contributor

Deploy preview for airbyte-docs ready!

✅ Preview
https://airbyte-docs-pmo6h9724-airbyte-growth.vercel.app

Built with commit e706e42.
This pull request is being automatically deployed with vercel-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant