[Feature Request] Allow skipping polling stage in AsyncRetriever when creation response provides download URL

[devin-ai-integration]

Summary

When configuring an asynchronous job in the Connector Builder, some APIs directly provide a download URL in the creation response, making the polling stage unnecessary. Currently, the polling_requester is a required field in the AsyncRetriever schema, forcing users to configure a polling endpoint even when it's not needed.

Problem

A customer reported struggling to configure an async job where the API directly provides a download URL during the creation stage. The current workaround involves:

1. Using the creation response variable in the Download tab
2. Adding a dummy URL (e.g., a fake GitHub status page URL) in the Polling tab to satisfy the required field

While this workaround functions, it's not an ideal solution and creates confusion for users.

Technical Details

Show/Hide Technical Analysis

The polling_requester is defined as a required field in the AsyncRetriever schema:

# declarative_component_schema.py:2353-2356
polling_requester: Union[CustomRequester, HttpRequester] = Field(
    ...,
    description="Requester component that describes how to prepare HTTP requests to send to the source API to fetch the status of the running async job.",
)

The AsyncHttpJobRepository class in http_job_repository.py always expects a polling response to extract download targets:

# http_job_repository.py:332-334
def _get_download_targets(self, job: AsyncJob) -> Iterable[str]:
    if not self.download_target_requester:
        url_response = self._polling_job_response_by_id[job.api_job_id()]

Proposed Solution

Make the polling_requester field optional in the AsyncRetriever schema and update the AsyncHttpJobRepository to handle the case where:

1. No polling is configured
2. The download URL is extracted directly from the creation response
3. The job is immediately marked as "completed" after creation

This would allow users to configure the download_target_extractor to use {{ creation_response.download_url }} (or similar) without needing a dummy polling endpoint.

Related Issues

• [Roadmap] Support Async data format in Connector Builder #55843 - [Roadmap] Support Async data format in Connector Builder (closed)
• Async Polling Streams in Airbyte Cloud Builder Fail with SocketTimeoutException After ~5 Minutes Despite Longer Polling Timeout #65127 - Async Polling Streams timeout issues (open)

Requested by

syed.khadeer@airbyte.io (@Airbyte-Support) on behalf of a customer

Investigation Link

https://app.devin.ai/sessions/7e6f6d6a45b44166bdbaf062154b7771

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and agent prompts for this issue.

• Create Plan

Examples

• Example 1
• Example 2

🔗 Related PRs

airbytehq/airbyte-python-cdk#211 - feat: Async Retriever add url_requester [merged] | airbyte_cdk/sources/declarative/declarative_component_schema.yaml:
"""

AI-generated summary of changes

The provided diff introduces a new property, url_requester, to the AsyncRetriever definition within the declarative_component_schema.yaml file. This property is defined as a requester component responsible for preparing HTTP requests to extract the URL from the polling response of a completed asynchronous job. The addition of this property enhances the functionality of the AsyncRetriever by enabling it to specifically handle the extraction of URLs from responses received after polling for job completion. The new property is structured similarly to existing requester components, maintaining consistency in how requests are constructed and executed.

The change does not involve any deletions or modifications to existing properties, ensuring that the overall structure and functionality of the AsyncRetriever remain intact while extending its capabilities.

Alterations to the declarations of exported or public entities

• Property added: url_requester in AsyncRetriever definition in airbyte_cdk/sources/declarative/declarative_component_schema.yaml
  """

---

airbyte_cdk/sources/declarative/models/declarative_component_schema.py:
"""

AI-generated summary of changes

The diff introduces a new class FlattenFields in the declarative_component_schema.py file, which is defined as a subclass of BaseModel. This class includes a type attribute set to the literal value "FlattenFields" and an optional flatten_lists attribute, which is a boolean that defaults to True. This attribute specifies whether to flatten lists or leave them as is. The addition of this class allows for a new transformation type within the schema, enabling the flattening of nested fields in the data structure.

Additionally, the KeysReplace class has undergone a reformatting of its examples attributes for better readability, with the examples now presented as lists across multiple lines instead of a single line. This change does not alter the functionality but improves the clarity of the examples provided for the old and new fields.

The previous definition of FlattenFields has been removed, indicating a restructuring of how this transformation is represented in the code.

Overall, the changes enhance the schema's capabilities by introducing a new transformation type while improving the formatting of existing examples for clarity.

Alterations to the declarations of exported or public entities

• Method added: class FlattenFields(BaseModel) in airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• Method signature updated: def KeysReplace(...) in airbyte_cdk/sources/declarative/models/declarative_component_schema.py (examples reformatted for clarity)
  """

---

airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py:
"""

AI-generated summary of changes

The diff introduces a new variable url_requester within the create_async_retriever method of the ModelToComponentFactory class. This variable is initialized by calling the _create_component_from_model method with the model.url_requester parameter, along with the existing decoder and config parameters. The url_requester is then included in the instantiation of the AsyncJobRepository, which allows the repository to handle URL extraction as part of the asynchronous job processing. This change enhances the functionality of the AsyncRetriever by integrating URL handling into the job retrieval process.

Alterations to the declarations of exported or public entities

• Variable added: url_requester in method create_async_retriever in class ModelToComponentFactory in airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py.
  """

---

airbyte_cdk/sources/declarative/requesters/http_job_repository.py:
"""

AI-generated summary of changes

The diff introduces a docstring to the AsyncHttpJobRepository class, providing additional context about its functionality. A new optional attribute url_requester is added, initialized to None, which is intended for handling additional requests to obtain download URLs when an identifier is provided by the polling_requester.

A new private method _get_download_url is introduced, which retrieves download URLs based on the job's API ID. If url_requester is not set, it uses the polling response stored in _polling_job_response_by_id. If url_requester is available, it constructs a StreamSlice using the polling response and sends a request to obtain the URLs. This method replaces the previous logic in the fetch_records method, which directly extracted URLs from the polling response.

The fetch_records method is modified to call the new _get_download_url method instead of directly extracting URLs from the polling response. This encapsulates the URL retrieval logic and allows for more flexible handling of the URL request process.

Overall, the changes enhance the functionality of the AsyncHttpJobRepository by introducing a mechanism for conditional URL retrieval based on the presence of a url_requester.

Alterations to the declarations of exported or public entities

• New attribute added: url_requester: Optional[Requester] in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py
• New method added: def _get_download_url(self, job: AsyncJob) -> Iterable[str] in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py
  """

---

airbyte_cdk/sources/declarative/requesters/README.md:
"""

AI-generated summary of changes

The changes introduce a sequence diagram for the AsyncHttpJobRepository, illustrating the interactions between various components involved in job management. The diagram includes participants such as CreationRequester, PollingRequester, UrlRequester, DownloadRetriever, AbortRequester, and DeleteRequester, with the UrlRequester, AbortRequester, and DeleteRequester marked as optional components. The flow begins with the initiation of job creation, followed by polling for job status. Depending on the job status, the diagram outlines the processes for requesting download URLs, retrieving report data, handling job failures with abort requests, and sending delete job requests. The diagram emphasizes the use of interpolation_context to obtain necessary context from responses, specifically from create_job_response and polling_job_response.

Alterations to the declarations of exported or public entities

• New documentation added: AsyncHttpJobRepository sequence diagram in airbyte_cdk/sources/declarative/requesters/README.md
  """
  fix: (CDK) (AsyncRetriever) - Improve UX on variable naming and interpolation airbyte-python-cdk#368 - fix: (CDK) (AsyncRetriever) - Improve UX on variable naming and interpolation [merged] | airbyte_cdk/sources/declarative/declarative_component_schema.yaml:
  """

AI-generated summary of changes

The changes in the declarative_component_schema.yaml file introduce three new fields: creation_response, polling_response, and download_target to the request_parameters section. These fields are defined with their respective descriptions and types, enhancing the schema to accommodate additional responses from asynchronous operations. The creation_response field captures the response from the creation_requester, while the polling_response field captures the response from the polling_requester. The download_target field specifies the URL received from the polling_requester when the job status is COMPLETED.

Additionally, the existing field urls_extractor has been renamed to download_target_extractor, reflecting a change in its purpose to focus on extracting the final download target URL from the polling response of an asynchronous job. The description of this field has been updated to clarify its role in fetching the final result URLs provided by the completed async job.

Overall, these modifications enhance the schema's capability to handle responses from asynchronous operations more effectively.

Alterations to the declarations of exported or public entities

• Field added: creation_response in definitions section of airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• Field added: polling_response in definitions section of airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• Field added: download_target in definitions section of airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• Field signature updated: urls_extractor in AsyncRetriever definition in airbyte_cdk/sources/declarative/declarative_component_schema.yaml → download_target_extractor in AsyncRetriever definition in airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• Field signature updated: url_requester in AsyncRetriever definition in airbyte_cdk/sources/declarative/declarative_component_schema.yaml → download_target_requester in AsyncRetriever definition in airbyte_cdk/sources/declarative/declarative_component_schema.yaml
  """

---

airbyte_cdk/sources/declarative/models/declarative_component_schema.py:
"""

AI-generated summary of changes

The changes made to the AsyncRetriever class in the declarative_component_schema.py file involve renaming two attributes: urls_extractor has been changed to download_target_extractor, and url_requester has been renamed to download_target_requester. The types of these attributes remain unchanged, with both still being unions of CustomRecordExtractor and DpathExtractor for download_target_extractor, and unions of CustomRequester and HttpRequester for download_target_requester. The descriptions associated with these attributes have also been retained, maintaining their original functionality. The overall structure and logic of the AsyncRetriever class remain intact, with no additional attributes or methods added or removed.

Alterations to the declarations of exported or public entities

• Variable renamed: urls_extractor: Union[CustomRecordExtractor, DpathExtractor] in class AsyncRetriever in airbyte_cdk/sources/declarative/models/declarative_component_schema.py → download_target_extractor: Union[CustomRecordExtractor, DpathExtractor] in class AsyncRetriever in airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• Variable renamed: url_requester: Optional[Union[CustomRequester, HttpRequester]] in class AsyncRetriever in airbyte_cdk/sources/declarative/models/declarative_component_schema.py → download_target_requester: Optional[Union[CustomRequester, HttpRequester]] in class AsyncRetriever in airbyte_cdk/sources/declarative/models/declarative_component_schema.py
  """

---

airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py:
"""

AI-generated summary of changes

The changes in the provided diff involve modifications to the _get_download_retriever method within the ModelToComponentFactory class. The previous instantiation of url_requester has been removed and replaced with download_target_requester, which is now created using the download_target_requester model instead of the url_requester model. This indicates a shift in the focus from handling URLs to handling download targets. Additionally, the urls_extractor component has been removed and replaced with download_target_extractor, which is instantiated using the download_target_extractor model. The parameters passed to the _create_component_from_model method remain consistent, indicating that the underlying logic for component creation is preserved, but the specific components being instantiated have changed. The job_repository instantiation has been updated to reflect these changes, utilizing the new download_target_requester and download_target_extractor components instead of their URL counterparts.

Alterations to the declarations of exported or public entities

• Method signature updated: def _get_download_retriever() -> SimpleRetrieverTestReadDecorator | SimpleRetriever in class ModelToComponentFactory in airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (no changes to the signature itself, but changes in the internal logic).
  """

---

airbyte_cdk/sources/declarative/requesters/README.md:
"""

AI-generated summary of changes

The changes in the documentation for the AsyncHttpJobRepository sequence diagram involve updates to terminology and references related to URL extraction and response contexts. The term url_requester has been replaced with download_target_requester, indicating a change in the component responsible for requesting download URLs when not provided. Similarly, urls_extractor has been updated to download_target_extractor, reflecting the new naming convention. The references to polling_job_response and create_job_response have been modified to polling_response and creation_response, respectively. These changes clarify the context from which responses are derived and maintain consistency in terminology throughout the sequence diagram.

The overall logic flow of the sequence diagram remains unchanged, but the updated terminology may influence user interpretation of the interactions between components, particularly regarding job status polling and URL requests.

Alterations to the declarations of exported or public entities

• No alterations to the declarations of exported or public entities were made in this diff.
  """

---

airbyte_cdk/sources/declarative/requesters/http_job_repository.py:
"""

AI-generated summary of changes

The changes in the AsyncHttpJobRepository class involve a comprehensive renaming and restructuring of attributes and methods related to URL handling. The urls_extractor attribute has been renamed to download_target_extractor, reflecting a shift in focus from URLs to download targets. The url_requester attribute has also been renamed to download_target_requester, indicating a similar change in the requester used for fetching download targets.

The fetch_records method has been modified to iterate over download_targets instead of urls, with the extra_fields dictionary now including download_target instead of url. This change alters how records are fetched and processed within the method.

In the _get_create_job_stream_slice method, a new variable creation_response is introduced to capture the JSON response from the job's creation request, which is then added to the extra_fields of the StreamSlice. The _get_download_url method has been renamed to _get_download_targets, and it now extracts the polling_response from the JSON response of the polling job. The partition dictionary in this method has been simplified to an empty dictionary.

Finally, the yield statement in the _get_download_targets method has been updated to use download_target_extractor for extracting records from the url_response. These changes reflect a reorganization of how download targets are managed within the job repository, enhancing clarity and consistency in the handling of download-related functionality.

Alterations to the declarations of exported or public entities

• Attribute updated: urls_extractor: DpathExtractor in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py → download_target_extractor: DpathExtractor in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py
• Attribute updated: url_requester: Optional[Requester] in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py → download_target_requester: Optional[Requester] in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py
• Method signature updated: def _get_download_url(self, job: AsyncJob) -> Iterable[str] in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py → def _get_download_targets(self, job: AsyncJob) -> Iterable[str] in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py
  """

---

airbyte_cdk/sources/declarative/requesters/http_requester.py:
"""

AI-generated summary of changes

The changes in the http_requester.py file involve two main modifications. First, in the __post_init__ method, the line that assigns backoff_strategies from self.error_handler.backoff_strategies has been updated to include a type ignore comment. This change indicates that the type checker should ignore potential type issues related to backoff_strategies, which may suggest that the type of backoff_strategies is uncertain or not explicitly defined in the context of the error handler.

Second, in the get_path method, additional logic has been introduced to update the interpolation context with extra fields from stream_slice. If stream_slice is not None and has the attribute extra_fields, these fields are unpacked and included in the kwargs dictionary. This enhancement allows for more dynamic path generation by incorporating extra fields into the context used for evaluating the path.

Overall, these changes enhance type handling and extend the functionality of the path generation process.

Alterations to the declarations of exported or public entities

• No alterations to the declarations of exported or public entities were made in this diff.
  """

---

unit_tests/sources/declarative/parsers/test_model_to_component_factory.py:
"""

AI-generated summary of changes

The changes in the test_create_async_retriever function involve the renaming of certain keys in the configuration dictionary. The key "urls_extractor" has been replaced with "download_target_extractor", indicating a shift in the naming convention for the extractor responsible for handling URLs. Additionally, the path for the polling_requester has been modified from a more complex expression that references stream_slice['create_job_response'].json()['id'] to a simpler reference to creation_response['id']. This change simplifies the path construction for the polling request. Furthermore, the paths for the download_requester, abort_requester, and delete_requester have been updated to use the new "download_target" variable instead of the previous reference to stream_slice['url']. Lastly, the assertion checking for urls_extractor has been updated to check for download_target_extractor, reflecting the earlier renaming.

Overall, these changes streamline the naming and path references within the async retriever configuration, enhancing clarity and consistency.

Alterations to the declarations of exported or public entities

• Key renamed: "urls_extractor" in the configuration dictionary → "download_target_extractor" in the configuration dictionary.
• Path updated: "path": "/v3/marketing/contacts/exports/{{stream_slice['create_job_response'].json()['id'] }}" in polling_requester → "path": "/v3/marketing/contacts/exports/{{creation_response['id'] }}" in polling_requester.
• Path updated: "path": "{{stream_slice['url']}}" in download_requester → "path": "{{download_target}}" in download_requester.
• Path updated: "path": "{{stream_slice['url']}}/abort" in abort_requester → "path": "{{download_target}}/abort" in abort_requester.
• Path updated: "path": "{{stream_slice['url']}}" in delete_requester → "path": "{{download_target}}" in delete_requester.
• Assertion updated: assert job_repository.urls_extractor → assert job_repository.download_target_extractor.
  """

---

unit_tests/sources/declarative/requesters/test_http_job_repository.py:
"""

AI-generated summary of changes

The changes in the diff involve modifications to the setUp method of the HttpJobRepositoryTest class, specifically altering the path parameters used in the HttpRequester instances and renaming a variable related to the DpathExtractor.

1. In the _polling_job_requester, the path was changed from:

   path=_EXPORT_PATH + "/{{stream_slice['create_job_response'].json()['id']}}"

   to:

   path=_EXPORT_PATH + "/{{creation_response['id']}}"

   This change simplifies the path extraction by directly referencing creation_response['id'] instead of extracting it from a JSON response.

2. In the _download_retriever, the path was modified from:

   path="{{stream_slice.extra_fields['url']}}"

   to:

   path="{{download_target}}"

   This change indicates a shift in how the download target URL is referenced, moving from a more complex extraction to a simpler variable reference.

3. The urls_extractor parameter in the AsyncHttpJobRepository instantiation was renamed to download_target_extractor, reflecting a change in the focus of the extractor from generic URL extraction to specifically targeting the download URL.

These changes collectively streamline the logic for handling job requests and responses, focusing on more direct references for paths and extraction.

Alterations to the declarations of exported or public entities

• Variable renamed: urls_extractor in class AsyncHttpJobRepository in unit_tests/sources/declarative/requesters/test_http_job_repository.py → download_target_extractor in class AsyncHttpJobRepository in unit_tests/sources/declarative/requesters/test_http_job_repository.py
  """

---

unit_tests/sources/declarative/test_concurrent_declarative_source.py:
"""

AI-generated summary of changes

The changes in the provided diff involve modifications to the configuration of the async_job_stream within the manifest. Specifically, the key urls_extractor has been renamed to download_target_extractor, which indicates a shift in the intended purpose of this extractor. The field_path remains unchanged, still pointing to ["urls"]. Additionally, the path for the polling_requester has been altered from a more complex expression that referenced stream_slice['create_job_response'].json()['id'] to a simpler reference to creation_response['id']. This change simplifies the path construction for the HTTP request, potentially improving clarity and maintainability.

The modifications do not introduce new functionality but rather refine existing configurations, focusing on clearer naming and simpler path resolution for the HTTP requester.

Alterations to the declarations of exported or public entities

• Key renamed: urls_extractor in the manifest for async_job_stream → download_target_extractor in the manifest for async_job_stream
• Path updated: "/async_job/{{stream_slice['create_job_response'].json()['id'] }}" in polling_requester → "/async_job/{{creation_response['id'] }}" in polling_requester
  """
  fix: (CDK) (AsyncRetriever) - Add polling_timeout optional field  airbyte-python-cdk#428 - fix: (CDK) (AsyncRetriever) - Add polling_timeout optional field [merged] | airbyte_cdk/sources/declarative/declarative_component_schema.yaml:
  """

AI-generated summary of changes

The changes in the provided diff involve modifications to the AsyncRetriever definition within the declarative_component_schema.yaml file. The description of the AsyncRetriever has been updated by removing the experimental disclaimer, which previously indicated that the interface might change and that such changes would not be considered breaking. The new description maintains the core functionality explanation of the AsyncRetriever, which is responsible for retrieving records by asynchronously sending requests.

Additionally, a new property named polling_job_timeout has been introduced. This property specifies the time in minutes after which a single asynchronous job should be considered timed out. It accepts either an integer or a string type and includes an interpolation context that allows for configuration-based values. The addition of this property enhances the functionality of the AsyncRetriever by providing a mechanism to handle job timeouts, which was not previously defined.

Overall, the changes reflect a refinement of the AsyncRetriever component's documentation and the introduction of a new timeout feature to improve its operational robustness.

Alterations to the declarations of exported or public entities

• Property added: polling_job_timeout in AsyncRetriever definition in airbyte_cdk/sources/declarative/declarative_component_schema.yaml
  """

---

airbyte_cdk/sources/declarative/models/declarative_component_schema.py:
"""

AI-generated summary of changes

The diff introduces a minor modification in the AddFields class and adds a new field to the AsyncRetriever class. Specifically, the condition field in the AddFields class has had its description updated to remove an extraneous comma at the end of the sentence. This change does not affect the functionality or logic of the code.

Additionally, a new optional field named polling_job_timeout has been added to the AsyncRetriever class. This field is of type Optional[Union[int, str]] and is intended to specify the time in minutes after which a single asynchronous job should be considered timed out. The addition of this field enhances the configuration options available for managing asynchronous job timeouts.

Overall, the changes involve a description correction and the introduction of a new timeout configuration for asynchronous operations.

Alterations to the declarations of exported or public entities

• New field added: polling_job_timeout: Optional[Union[int, str]] in class AsyncRetriever in airbyte_cdk/sources/declarative/models/declarative_component_schema.py
• Field description updated: condition: Optional[str] in class AddFields in airbyte_cdk/sources/declarative/models/declarative_component_schema.py (description changed from "Fields will be added if expression is evaluated to True.," to "Fields will be added if expression is evaluated to True.")
  """

---

airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py:
"""

AI-generated summary of changes

The diff introduces a new function _get_job_timeout() within the _get_download_retriever() method. This function computes a job timeout based on a user-defined polling timeout from the model. It evaluates the polling_timeout and converts it into a datetime.timedelta object. If no user-defined timeout is provided, it defaults to a test read timeout of 15 minutes and a sync timeout of 60 minutes for non-connector builders. The function returns the test read timeout if connector builder messages are emitted; otherwise, it returns the default sync timeout.

Additionally, the job_timeout parameter is added to the instantiation of AsyncHttpJobRepository, which now receives the value returned from _get_job_timeout(). The import statement for ConnectionDefinition is removed from the top of the file, leaving only the import for Config.

Overall, these changes enhance the timeout handling logic for job processing within the asynchronous job repository, allowing for more flexible timeout configurations based on user input.

Alterations to the declarations of exported or public entities

• Method added: def _get_job_timeout() -> datetime.timedelta in function _get_download_retriever in airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
• Variable removed: ConnectionDefinition from import statement in airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
  """

---

airbyte_cdk/sources/declarative/requesters/http_job_repository.py:
"""

AI-generated summary of changes

The changes introduce a new optional attribute job_timeout to the AsyncHttpJobRepository class, which is initialized to None and is annotated as Optional[timedelta]. This attribute is intended to specify a timeout for job completion. Additionally, the logging description in the _start_job_and_validate_response method has been updated to reflect this new attribute. The previous static description "Create the server-side async job." has been replaced with a dynamic string that includes the value of self.job_timeout, stating "Create the server-side async job. Timeout after: {self.job_timeout}". This modification enhances the logging output by providing more context regarding the job's timeout setting during the job creation process.

Alterations to the declarations of exported or public entities

• New attribute added: job_timeout: Optional[timedelta] = None in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py
  """

---

airbyte_cdk/sources/declarative/retrievers/async_retriever.py:
"""

AI-generated summary of changes

The changes involve the removal of a deprecation warning for the AsyncRetriever class. Specifically, the @deprecated decorator, which indicated that the class was experimental and advised caution in its use, has been deleted. This alteration modifies the class's status from being marked as experimental to no longer having any explicit warning regarding its experimental nature. The rest of the class definition and its functionality remain unchanged.

Alterations to the declarations of exported or public entities

• Decorator removed: @deprecated("This class is experimental. Use at your own risk.", category=ExperimentalClassWarning) from class AsyncRetriever in airbyte_cdk/sources/declarative/retrievers/async_retriever.py
  """
  feat(AsyncRetriever): slice information during the polling request, Support creation_response interpolation in body airbyte-python-cdk#541 - feat(AsyncRetriever): slice information during the polling request, Support creation_response interpolation in body [merged] | airbyte_cdk/sources/declarative/requesters/http_job_repository.py:
  """

AI-generated summary of changes

The method _get_create_job_stream_slice was refactored to directly return a StreamSlice object instead of first assigning it to a local variable. The returned StreamSlice now uses the partition and cursor_slice values directly from job.job_parameters() instead of empty dictionaries. The extra_fields dictionary is constructed by merging a copy of job.job_parameters().extra_fields with a new key "creation_response" whose value is the creation response interpolation context for the job. This replaces the previous approach where extra_fields only contained the "creation_response" key without including the original extra_fields from the job parameters.

Alterations to the declarations of exported or public entities

• No changes to the signatures of exported or public entities were made. The change is internal to the _get_create_job_stream_slice method implementation only.
  """

---

airbyte_cdk/sources/declarative/requesters/request_options/interpolated_nested_request_input_provider.py:
"""

AI-generated summary of changes

The change replaces the manual construction of a dictionary containing stream_slice and next_page_token with a call to the utility function get_interpolation_context, passing the same two parameters. This modifies how the interpolation context dictionary is created before being passed as keyword arguments to the _interpolator.eval method. The overall logic of passing these parameters for interpolation remains the same, but the context dictionary is now generated via a centralized helper function instead of inline dictionary construction.

Alterations to the declarations of exported or public entities

None
"""

---

airbyte_cdk/sources/declarative/requesters/request_options/interpolated_request_input_provider.py:
"""

AI-generated summary of changes

The change replaces a manually constructed dictionary containing stream_slice and next_page_token with a call to the utility function get_interpolation_context passing the same two parameters. This modifies how the interpolation context is built before passing it as keyword arguments to the interpolator's eval method. The rest of the logic, including the evaluation of the interpolated value and filtering out keys with None values, remains unchanged.

Alterations to the declarations of exported or public entities

None.
"""

---

unit_tests/sources/declarative/requesters/test_http_job_repository.py:
"""

AI-generated summary of changes

The test setup in HttpJobRepositoryTest was refactored by extracting the creation of the AsyncHttpJobRepository instance into a new helper method _create_async_job_repository. This method consolidates the construction of the HttpRequester instances for job creation, polling, and downloading, along with the SimpleRetriever and associated components, using a shared mocked MessageRepository and a predefined DefaultErrorHandler instance _ERROR_HANDLER. The original verbose setup code in setUp was replaced by a single call to this helper.

A new test method test_when_update_jobs_status_then_allow_access_to_stream_slice_information was added. It verifies that the repository correctly handles stream slice information in the polling request path template, by creating a custom polling HttpRequester that includes stream_slice['path'] in the URL path. The test mocks the HTTP response for the job status and asserts that the job status updates to COMPLETED as expected.

The constant _ERROR_HANDLER was introduced at the module level to reuse a single DefaultErrorHandler instance with empty config and parameters.

No changes were made to the existing test logic or assertions besides the addition of the new test and the refactor of setup code.

Alterations to the declarations of exported or public entities

• Method added: def _create_async_job_repository(self, polling_job_requester: Optional[HttpRequester] = None) -> AsyncHttpJobRepository in class HttpJobRepositoryTest in unit_tests/sources/declarative/requesters/test_http_job_repository.py
• Method added: def test_when_update_jobs_status_then_allow_access_to_stream_slice_information(self) -> None in class HttpJobRepositoryTest in unit_tests/sources/declarative/requesters/test_http_job_repository.py
  """
  feat: additional interpolation contexts for download step of async job airbyte-python-cdk#757 - feat: additional interpolation contexts for download step of async job [merged] | airbyte_cdk/sources/declarative/declarative_component_schema.yaml:
  """

AI-generated summary of changes

• Top-level definitions: Updated description of download_target_extractor from fetching the url where the file is located (applied per-record, not on the HTTP response) to fetching the final result urls provided by the completed/finished/ready async job.
• AsyncRetriever: The download_target_extractor field is no longer required (removed from the required list), though the property remains defined.
• AsyncRetriever: The download_target_extractor description within the property changed from fetching the final result urls provided by the completed/finished/ready async job to fetching the information needed to download the completed job from the polling HTTP response.
• AsyncRetriever: No other structural changes to the retriever wiring; the property continues to reference DpathExtractor or CustomRecordExtractor as its anyOf.

Alterations to the declarations of exported or public entities

• Definition updated: download_target_extractor description in the top-level declarations of airbyte_cdk/sources/declarative/declarative_component_schema.yaml
• Public entity updated: AsyncRetriever - download_target_extractor removed from the required fields
• Public entity updated: download_target_extractor description inside AsyncRetriever changed to indicate it fetches information from the polling response instead of final URLs
  Estimated code review effort: Medium

Estimated code review effort: Medium
"""

---

airbyte_cdk/sources/declarative/models/declarative_component_schema.py:
"""

AI-generated summary of changes

• Changes the AsyncRetriever field download_target_extractor from a required union type to an optional union type, introducing a default of None. Updates the field’s description to reflect downloading from the polling HTTP response rather than final URLs from a completed job. This alters validation and availability of the field at instantiation, affecting downstream logic that assumed the field would always be present.

Alterations to the declarations of exported or public entities

• Field signature updated: download_target_extractor in class AsyncRetriever in airbyte_cdk/sources/declarative/models/declarative_component_schema.py → from
  download_target_extractor: Union[DpathExtractor, CustomRecordExtractor] = Field( ..., description="Responsible for fetching the final result urls provided by the completed / finished / ready async job.", )
  to
  download_target_extractor: Optional[Union[DpathExtractor, CustomRecordExtractor]] = Field( None, description="Responsible for fetching the information needed to download the completed job from the polling HTTP response.", )

Estimated code review effort: Minor

Estimated code review effort: Medium
"""

---

airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py:
"""

AI-generated summary of changes

• Added a precondition: when using a download_target_requester, require download_target_extractor to be provided; otherwise raise ValueError with message "download_target_extractor required if using a download_target_requester".
• In create_async_retriever, download_target_extractor is now created conditionally: only instantiate it if model.download_target_extractor is truthy; otherwise assign None. This propagates to the AsyncJobRepository as a possibly None download_target_extractor, altering runtime wiring.
• This changes the runtime wiring of AsyncHttpJobRepository to accommodate an optional download_target_extractor.

Alterations to the declarations of exported or public entities

• None

Estimated code review effort: Medium
"""

---

airbyte_cdk/sources/declarative/requesters/README.md:
"""

AI-generated summary of changes

• Added documentation for a new Download Target and Download Requester flow. Specifically, states that creation_response and polling_response contexts are always available during the job download step, and that a download_target context is produced by download_target_extractor and exposed to the job download step.
  • If download_target_requester is not provided, download_target_extractor will derive URLs from polling_response.
  • If download_target_requester is provided, an extra request fetches job download targets and download_target_extractor operates on that response.
• Introduced a new "Some important considerations" section:
  • If download_target_extractor and download_target_requester are not defined, a single job download request is made without the download_target context.
  • The download_target_extractor is required (not optional) if using a download_target_requester.
• Updated the AsyncHttpJobRepository sequence diagram:
  • The DownloadRetriever’s Retrieve report data now uses interpolation_context: download_target, creation_response, polling_response (instead of only url).
• Removed old guidance lines that previously stated:
  • When download_target_requester is not provided, download_target_extractor will get URLs from polling_response.
  • Interpolation_context guidance such as obtaining from stream_slice for certain contexts.
• No changes to declarations of exported/public entities.
• Estimated code review effort: Low.

Alterations to the declarations of exported or public entities

• No changes to exported/public entity declarations.
  """

---

airbyte_cdk/sources/declarative/requesters/http_job_repository.py:
"""

AI-generated summary of changes

• Public field signature updated: download_target_extractor in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py changed from DpathExtractor to Optional[DpathExtractor], allowing absence of the extractor.

• fetch_records loop and stream slice enrichment:

  • Loop variable renamed from target_url to download_target.
  • ExtraFields now include:
    • "download_target": download_target
    • "creation_response": self._get_creation_response_interpolation_context(job)
    • "polling_response": self._get_polling_response_interpolation_context(job)

• _get_download_targets refactor (multi-path logic):

  • Added docstring describing targets functionality.
  • Fallback behavior: if neither download_target_extractor nor download_target_requester is provided, log a debug message and yield an empty string (""), enabling a single download request without a download_target.
  • If no extractor but a requester is provided, raise a config error requiring a download_target_extractor when using a download_target_requester.
  • If an extractor is present and a download_target_requester is provided, build and use a request to obtain download targets via the requester; validate response and raise on failure.
  • If an extractor is present and no requester is provided, derive targets from the polling response.
  • In non-fallback paths, yield targets by calling download_target_extractor.extract_records(url_response).

• Additional structural/docstring updates:

  • Introduced a debug log in the fallback case.
  • Expanded explanatory comments and docstrings for _get_download_targets and related flow.

• Overall behavior: supports a fallback when both extractor and requester are absent, enriches stream slices with creation/polling contexts, and adjusts the target extraction flow accordingly.

Review tag

Alterations to the declarations of exported or public entities

• Field signature updated: download_target_extractor in class AsyncHttpJobRepository in airbyte_cdk/sources/declarative/requesters/http_job_repository.py → from DpathExtractor to Optional[DpathExtractor]

Estimated code review effort: High
"""

👤 Suggested Assignees

• bazarnov
• artem1205
• aldogonzalez8
• dbgold17

---

🧪 Issue enrichment is currently in early access.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

[octavia-squidington-iii]

Zendesk ticket #15403 has been linked to this issue.