Arax pathfinder by mohsenht · Pull Request #64 · BioPack-team/shepherd

mohsenht · 2026-01-21T16:17:24Z

Please review this pull request.

codecov · 2026-01-21T16:19:59Z

Codecov Report

❌ Patch coverage is 43.52518% with 157 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.79%. Comparing base (932a569) to head (6fbd183).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
workers/arax_pathfinder/worker.py	57.57%	66 Missing and 4 partials ⚠️
shepherd_utils/data_download.py	26.76%	51 Missing and 1 partial ⚠️
workers/arax/worker.py	0.00%	35 Missing ⚠️

Files with missing lines	Coverage Δ
shepherd_utils/config.py	`95.23% <100.00%> (+0.34%)`	⬆️
shepherd_utils/inject_shepherd_arax_provenance.py	`100.00% <ø> (ø)`
shepherd_utils/reclaim.py	`77.41% <ø> (ø)`
workers/arax/worker.py	`0.00% <0.00%> (ø)`
shepherd_utils/data_download.py	`55.73% <26.76%> (-34.27%)`	⬇️
workers/arax_pathfinder/worker.py	`57.57% <57.57%> (ø)`

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0450fde...6fbd183. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

maximusunc · 2026-02-04T16:19:48Z

I tried running a query through the ARAX pathfinder and it's unclear what happened. I got these logs:

arax_pathfinder  | 2026-02-04T16:02:05.361576: DEBUG: lookup map not here! /tmp/biolink/biolink_lookup_map_4.2.5_v5.pickle
arax_pathfinder  | 2026-02-04T16:02:07.267560: INFO: Building local Biolink 4.2.5 ancestor/descendant lookup map because one doesn't yet exist
arax_pathfinder  | [2026-02-04 16:02:07,299: INFO/shepherd.arax.pathfinder.e3355332.00527f4e]: Model release date: 12/01/2025
arax_pathfinder  | [2026-02-04 16:02:07,299: INFO/shepherd.arax.pathfinder.e3355332.00527f4e]: Finding paths process has started
arax_pathfinder  | [2026-02-04 16:02:07,300: INFO/shepherd.arax.pathfinder.e3355332.00527f4e]: Expanding CHEBI:45783
arax_pathfinder  | [2026-02-04 16:02:07,301: INFO/shepherd.arax.pathfinder.e3355332.00527f4e]: Expanding MONDO:0004979

but nothing else before it was timed out after 5 minutes. I sent it Imatinib->Asthma.

mohsenht · 2026-02-04T18:52:46Z

Hi @maximusunc,

I changed the parameters to make it faster for now. I will get back to Shepherd-pathfinder and check it probably next week to figure out what the problem is.

…hfinder

maximusunc

When testing with Imatinib->Asthma, your Pathfinder is returning 0 paths. Is this intended?

maximusunc · 2026-02-16T17:38:47Z

+    try:
+        start = time.perf_counter()
+        logger.info("Starting pathfinder.get_paths()")
+        result, aux_graphs, knowledge_graph = pathfinder.get_paths(


I'm not sure if your pathfinder code is asynchronous or not, but this call is blocking and so your pathfinder implementation can only handle one query at a time. Is this intended?

Hi @maximusunc

Could you please provide me your json query that you sent and got 0 paths?

Here is my query and I got result for this one.

{ "message": { "query_graph": { "nodes": { "n0": { "ids": [ "CHEBI:31690" ] }, "n1": { "ids": [ "MONDO:0004979" ] } }, "paths": { "p0": { "subject": "n0", "object": "n1", "predicates": [ "biolink:related_to" ], "constraints": [] } } } } }

It is now can handle multiple queries.

So now we have the opposite effect here. It looks like now we're trying to handle every query that comes in, and running this locally, the RAM and CPU usage of this worker shot way up and hit my docker limits. I think we need to tune this so that CPU and RAM stay reasonable. What do we think is reasonable @mohsenht @dkoslicki ?

I'm also a little concerned with the query that you sent that does return results. That Chebi curie is Imatinib, but it is not normalized. Are you doing some normalization in your pathfinder code somewhere?

The worker pool capped at 4, each query takes under 1GB of RAM.
Since requests already take 2 to 4 minutes, I'm worried that reducing the workers to save resources will make end users wait way too long.

I'm not sure I follow. What worker pool are you talking about? And how Shepherd is set up, it is made to horizontally scale, so if one arax_pathfinder worker can't handle all the requests coming in, Shepherd can just spin up another one. So when we build these individual workers, we should make sure that we understand where the bottlenecks are and pick a reasonable threshold for what it should be able to handle as a single worker, and then we can duplicate them at the kubernetes level to handle more load.

Ah, that makes sense regarding Shepherd's horizontal scaling. The local resource spike is coming from num_cores = min(multiprocessing.cpu_count(), 4) inside the pathfinder library.

I added that multiprocessing specifically to speed up path expansion and keep query times down to 2-4 minutes. If we restrict each worker to 1 core to reduce its footprint, individual queries will take significantly longer.

We have a trade-off: Are we okay with longer end-user wait times so Shepherd can scale smaller workers? Or should we set a higher baseline per worker (e.g., 4 cores) to keep queries fast, and let Shepherd scale those larger pods?

maximusunc · 2026-02-26T15:15:29Z

I ran some tests last night and ran into some issues. I was able to run your query and get back results, but then I tried sending 5 concurrent queries and while they all fired off, I got this error for all of them:

arax_pathfinder  | requests.exceptions.ConnectionError: HTTPSConnectionPool(host='kg2cplover3.rtx.ai', port=9990): Max retries exceeded with url: /query (Caused by NewConnectionError("HTTPSConnection(host='kg2cplover3.rtx.ai', port=9990): Failed to establish a new connection: [Errno 111] Connection refused"))

And then I tried backing off and just sending one query and got this error:

arax_pathfinder  | [2026-02-26 02:18:32,536: ERROR/shepherd.arax.pathfinder.d18b72ad.f07114e6]: Path MONDO:0004979MONDO:0011786 raised an exception: MySQL connection failed: 2003 (HY000): Can't connect to MySQL server on 'arax-databases-mysql.rtx.ai:3306' (111)
arax_pathfinder  | [2026-02-26 02:18:34,166: ERROR/shepherd.arax.pathfinder.d18b72ad.f07114e6]: Path CHEBI:31690NCBIGene:1544 raised an exception: MySQL connection failed: 2003 (HY000): Can't connect to MySQL server on 'arax-databases-mysql.rtx.ai:3306' (111)
arax_pathfinder  | [2026-02-26 02:18:34,186: ERROR/shepherd.arax.pathfinder.d18b72ad.f07114e6]: PathFinder failed to find paths between on and sn. Error message is: MySQL connection failed: 2003 (HY000): Can't connect to MySQL server on 'arax-databases-mysql.rtx.ai:3306' (111)

Now this doesn't seem to be an issue with Shepherd but more coming from these external services. So my follow up questions are:
Can these external services handle concurrent queries? If they can't, then your Pathfinder shouldn't either. Is your Pathfinder heavily CPU-bound? If so, then we will want to set up some multi-processing potentially.

mohsenht · 2026-02-26T15:30:26Z

PloverDB Concurrency: The error came from PloverDB, which is actually designed to handle thousands of requests in parallel for Pathfinder and other services. The KG2 team is currently working hard on its stability.

Pathfinder Performance: Pathfinder is both CPU-bound and IO-bound. It already uses multiprocessing in its core code to calculate rankings and expand nodes while building paths and trees.

The subsequent failure for the single query shows that the database connection to arax-databases-mysql was also down. I will ping KG2 team for this one.

Thanks Max

maximusunc · 2026-03-10T13:15:33Z

Hey @mohsenht is this ready for another review?

…hfinder # Conflicts: # workers/arax/worker.py

mohsenht · 2026-03-11T00:51:44Z

Hi @maximusunc

yes, ready to review

maximusunc

With these changes, I was able to start a pathfinder query, but I still got an error when connecting to kg2cplover3.rtx.ai.

mohsenht · 2026-03-11T16:12:52Z

Hi @maximusunc,

Since these changes mostly from your work in the main branch, could you please resolve these conflicts yourself?

mohsenht · 2026-03-11T18:24:31Z

@maximusunc

I ran the branch and tested it for this query and it worked.

POST:
http://localhost:5439/arax/query

{
    "message": {
        "query_graph": {
            "nodes": {
                "n0": {
                    "ids": [
                        "CHEBI:31690"
                    ]
                },
                "n1": {
                    "ids": [
                        "MONDO:0004979"
                    ]
                }
            },
            "paths": {
                "p0": {
                    "subject": "n0",
                    "object": "n1",
                    "predicates": [
                        "biolink:related_to"
                    ],
                    "constraints": []
                }
            }
        }
    }
}

maximusunc · 2026-03-27T14:43:00Z

Here's some stats from some more testing:

I'm running Imatinib to Asthma
For Imatinib, if I send CHEBI:45783 (the normalized curie), I get 0 paths back, but if I send CHEBI:31690, I get 500 paths. Seems to me like we want to fix this normalization issue before merging in? @dkoslicki for awareness since we also have that slack convo going
When sending a single pathfinder query, I'm seeing RAM usage go from a baseline 1GB all the way up to 8GB. Can you confirm that this is intended? Is it loading the whole XGBoost model into memory at query time?

mohsenht · 2026-03-27T17:44:35Z

Since pathfinder is using ARAX node synonymizer and this synomnymizer differs from Node Norm, you got 0 results.
For the memory issue you mentioned I will investigate to check and try to use less memory and let you know.

…hfinder # Conflicts: # workers/arax/worker.py

…hfinder # Conflicts: # shepherd_utils/config.py

…hfinder

The pathfinding search ran via asyncio.to_thread, so its GIL-bound work competed with the event loop -- and TASK_LIMIT was 100, meaning up to 100 concurrent searches. The heartbeat is a coroutine on that same loop pinging every 5s against a 15s freshness window, so under load a live worker read as dead: reclaim_orphaned could XCLAIM a still-running task to a peer and duplicate it, and a stall past 60s tripped LoopWatchdog into os._exit(1). Restructure to match aragorn_score / arax_rank: - arax_pathfinder_task is a sync process-pool entrypoint taking only the query/response ids. It loads, searches, rehydrates, and saves entirely in a spawned child, so neither the search nor the message encode touches the parent's loop or heap. rehydrate becomes sync (httpx.Client) since it now runs in the child. - process_task delegates to the shared run_task_lifecycle via a pool closure, replacing the hand-rolled copy that used tracer.start_span instead of start_as_current_span (so httpx spans never nested under the task span) and never recorded exceptions on the span. - poll_for_tasks builds a ProcessPoolManager sized by resolve_pool_workers and passes max_workers to get_tasks; TASK_LIMIT drops 100 -> 10. Failures now reach the lifecycle instead of being swallowed. The three query-graph validation failures returned `message, 500`, a value the caller discarded, so the task was wrapped up as a success with no message ever written to response_id -- leaving the next worker to KeyError on it. Search failures were saved as a non-TRAPI {"status": "error"} blob that then flowed down the workflow. Both now raise and route to finish_query with ERROR. Also: - The blocked-concept list is fetched once at startup via ensure_http_files_dataset (temp file + atomic rename) rather than with a per-task requests.get racing on the same relative path; children parse it once and memoize, as they do the BiolinkHelper descendants. - Add "arax.pathfinder" to PER_STREAM_MIN_IDLE_SEC. It was absent, so its reclaim floor was the 30s fast default; 360s sits above pool_task_timeout_sec. - Dockerfile gets PYTHONFAULTHANDLER=1 so a child hung during import still dumps a stack when ProcessPoolManager SIGABRTs it. - Fix rehydrate's no-op `.replace("query", "rehydrate")` on an already-correct URL, and its references to a possibly-unbound `res` in the error handlers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H51TTxq1F6o6yZSP1C86MZ

…r-heartbeat-5ahuhf Claude/async arax pathfinder heartbeat 5ahuhf

maximusunc

Looks good and passes the same number of automated tests!

mohsenht added 3 commits January 20, 2026 10:16

Using Pathfinder package with local sqlite files

cba10f7

Using Pathfinder package with mysql server

5539e3d

Settings for arax pathfinder

cf94d6d

mohsenht requested a review from maximusunc January 21, 2026 16:17

dkoslicki reviewed Jan 21, 2026

View reviewed changes

Comment thread workers/arax/worker.py Outdated

dkoslicki reviewed Jan 21, 2026

View reviewed changes

Comment thread workers/arax_pathfinder/worker.py Outdated

mohsenht added 4 commits January 21, 2026 13:55

Black style errors

4999dff

Black style errors

767e51b

New pathfinder package release update.

540f220

Inject shepherd-arax provenance in all edge sources field

cd4cf5a

Temporary faster pathfinder by decreasing parameters

ce1a5e6

mohsenht added 2 commits February 11, 2026 11:09

Merge branch 'main' of github.com:BioPack-team/shepherd into arax-pat…

7cd663a

…hfinder

Arax Pathfinder tested with 4 hops

8ec57a9

maximusunc requested changes Feb 16, 2026

View reviewed changes

Async Arax Pathfinder

82852c4

mohsenht added 2 commits March 2, 2026 22:47

Pathfinder package updated

6b537e5

Pathfinder package updated

9919306

mohsenht added 2 commits March 10, 2026 20:40

Merge branch 'main' of github.com:BioPack-team/shepherd into arax-pat…

ab7049e

…hfinder # Conflicts: # workers/arax/worker.py

resolved conflicts

4771f90

mohsenht added 2 commits March 10, 2026 20:55

resolved conflicts

107614b

resolved conflicts

268d226

maximusunc requested changes Mar 11, 2026

View reviewed changes

Comment thread workers/arax/worker.py Outdated

Comment thread workers/arax/worker.py Outdated

Comment thread workers/arax_pathfinder/worker.py Outdated

Comment thread workers/arax_pathfinder/worker.py Outdated

PRUNE more

70758cb

mohsenht and others added 19 commits April 30, 2026 13:53

Merge branch 'main' of github.com:BioPack-team/shepherd into arax-pat…

b048766

…hfinder # Conflicts: # workers/arax/worker.py

Merge branch 'main' of github.com:BioPack-team/shepherd into arax-pat…

60b7d4a

…hfinder # Conflicts: # workers/arax/worker.py

Async call

a7ab6be

Merge branch 'main' of github.com:BioPack-team/shepherd into arax-pat…

d344b95

…hfinder # Conflicts: # shepherd_utils/config.py

Merge branch 'main' of github.com:BioPack-team/shepherd into arax-pat…

ebff3e6

…hfinder

ARAX Pathfinder Package 2.4.3. Adaptable with Retriever

a3eb40d

Auto download sqlite files for developers

7bc9ba2

Merge branch 'main' of github.com:BioPack-team/shepherd into arax-pat…

a3f35df

…hfinder

redundant plover_url config removed.

b823e96

ssh path generalized for all developers

5195035

readme correction

c699d27

Fetch sqlite files over Https instead of scp

cbd9ad5

arax_pathfinder_dbs git ignored

7a0730f

Run black

d41090f

Fix test import

ff282dc

Add script for testing against pathfinder

78d17af

Set arax pathfinder resources

a782c5f

Merge pull request #137 from BioPack-team/claude/async-arax-pathfinde…

e6ff46d

…r-heartbeat-5ahuhf Claude/async arax pathfinder heartbeat 5ahuhf

maximusunc previously approved these changes Aug 12, 2026

View reviewed changes

Add arax pathfinder to release

6fbd183

maximusunc dismissed their stale review via 6fbd183 August 12, 2026 19:16

maximusunc self-requested a review August 12, 2026 19:21

maximusunc approved these changes Aug 12, 2026

View reviewed changes

maximusunc merged commit 8a8dfb4 into main Aug 12, 2026
2 checks passed

maximusunc deleted the arax-pathfinder branch August 12, 2026 19:22

Uh oh!

Conversation

mohsenht commented Jan 21, 2026

Uh oh!

codecov Bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

maximusunc commented Feb 4, 2026

Uh oh!

mohsenht commented Feb 4, 2026

Uh oh!

maximusunc left a comment

Choose a reason for hiding this comment

Uh oh!

maximusunc Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

mohsenht Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

mohsenht Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohsenht Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maximusunc Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

maximusunc Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

mohsenht Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

maximusunc Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

mohsenht Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

maximusunc commented Feb 26, 2026

Uh oh!

mohsenht commented Feb 26, 2026

Uh oh!

maximusunc commented Mar 10, 2026

Uh oh!

mohsenht commented Mar 11, 2026

Uh oh!

maximusunc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mohsenht commented Mar 11, 2026

Uh oh!

mohsenht commented Mar 11, 2026

Uh oh!

maximusunc commented Mar 27, 2026

Uh oh!

mohsenht commented Mar 27, 2026

Uh oh!

maximusunc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

codecov Bot commented Jan 21, 2026 •

edited

Loading

mohsenht Feb 18, 2026 •

edited

Loading

mohsenht Feb 18, 2026 •

edited

Loading