Making a GraphQL Toolset (just like OpenAPI Toolset) and failing deployment to agent engine

[rushiiltecho]

Despite the agent working on the local playground, the deployment is failing through Agent Engine.
Here is an attachment document with the description of this issue:
https://docs.google.com/document/d/1mfKr-VkjZ11JEo0OK19l8Gv1LbWXKgAILMUHmr_s9tA/edit?usp=sharing

Refer to the following github repo for trying to recreate the issue:
https://github.com/Ashwin-Python/linear_graphql_adk_agent

Since Linear is based on the GraphQL Ecosystem of API Documentation, we came up with this approach and were able to make it work for the local playground; Though, upon trying deployment, it is failing "silently" without any errors except, it is giving:

google.api_core.exceptions.InvalidArgument: 400 Reasoning Engine resource [projects/1023299171373/locations/us-central1/reasoningEngines/5158851382920347648] failed to start and cannot serve traffic. Please refer to our documentation (https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/troubleshooting/deploy) for checking logs and other troubleshooting tips. 3: Reasoning Engine resource [projects/1023299171373/locations/us-central1/reasoningEngines/5158851382920347648] failed to start and cannot serve traffic. Please refer to our documentation (https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/troubleshooting/deploy) for checking logs and other troubleshooting tips.

We tried deploying Google Drive Agent (supported by google-adk internally) with the same deployment script to understand if the issue is with the deployment script but it turned out to be correctly deploying. Upon more searching through the documentation and other sources, it was a plausible pickling/unpickling issue on Agent Engine side but nothing concrete in the code suggests that (neither do the Logs suggest any issues) before it breaks and shows the above error message.

Would be happy to give more context, if required. This is an urgent requirement and any help would be appreciated from the community as well

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: The error Reasoning Engine ... failed to start when deploying to Agent Engine, despite working locally, is very often caused by a serialization (pickling) issue with a custom tool. The agent and all its components must be serializable to be deployed.

---

Hello! Thank you for the detailed question and for sharing your repository. It's great to see the community building new toolsets like the one you've created for GraphQL.

The error you're encountering is a common hurdle when moving from a local environment to a managed deployment like Vertex AI Agent Engine. Based on the symptoms you've described, your suspicion about a pickling/unpickling issue is very likely correct.

The Likely Cause: Serialization (Pickling)

When you deploy to Agent Engine, the ADK framework needs to serialize (or "pickle") your entire agent object, including all its tools and their dependencies, to send it to the Vertex AI environment. On the server, it deserializes ("unpickles") the agent to run it. If any part of your GraphQLToolset—or any object it holds onto, like a GraphQL client instance—is not serializable, the process will fail on the server side, resulting in the generic "failed to start" error you are seeing.

This explains why it works locally (where nothing needs to be serialized for deployment) but fails on Agent Engine.

How to Troubleshoot and Fix

Here are a few steps you can take to isolate and resolve the issue:

1. Explicitly Test for Pickling Issues

You can quickly check if your agent object is pickleable with a short local script. This is the most direct way to confirm the hypothesis. Add a temporary file (e.g., check_pickle.py) to your project:

import pickle
from agent import root_agent # Or however you instantiate your main agent

# This will raise an error if the agent or any of its tools are not pickleable.
try:
    pickled_agent = pickle.dumps(root_agent)
    print("✅ Agent was successfully pickled!")

    unpickled_agent = pickle.loads(pickled_agent)
    print("✅ Agent was successfully unpickled!")

except Exception as e:
    print(f"❌ Pickling failed: {e}")

Run this script. If it fails, the error message will point you directly to the object that cannot be serialized.

2. Check GraphQL Client Initialization

A common cause for pickling errors is holding onto non-serializable objects (like network connections or complex client objects with locks) as attributes of your tool or toolset class.

• Bad Pattern (likely to fail pickling):

  from gql import Client
  from gql.transport.aiohttp import AIOHTTPTransport
  
  class MyGraphQLTool:
      def __init__(self):
          # Storing the client instance can cause pickling issues
          self.client = Client(transport=AIOHTTPTransport(url="..."))
  
      def run(self, query):
          return self.client.execute(query)

• Good Pattern (pickle-safe):

  from gql import Client
  from gql.transport.aiohttp import AIOHTTPTransport
  
  class MyGraphQLTool:
      def run(self, query):
          # Initialize the client inside the method that uses it
          client = Client(transport=AIOHTTPTransport(url="..."))
          return client.execute(query)

  Instantiating the client inside the tool's execution method ensures it doesn't need to be pickled as part of the tool's state.

3. Use Verbose Deployment Logs

When you run your deployment script, ensure you are using the highest log verbosity. If you are using adk deploy agent_engine, you can add the --log_level DEBUG flag. This might provide more context about what's failing during the startup process.

4. Simplify and Isolate

Try creating a "dummy" version of your GraphQLToolset that has the same structure but where the tools do nothing (or just return a fixed string). If this dummy version deploys successfully, you can then add back the real logic piece by piece to find the exact line or dependency that is causing the problem.

This is a challenging but solvable issue. Your work on a GraphQL toolset is a valuable contribution, and I hope these steps help you get it deployed successfully!

[rushiiltecho]

We have tried with this script and it turned out to not be a pickling issue, and yes to the best of our knowledge we're using these pickle-safe good patterns.