LLM-Driven Automated Testing (10): How Healenium's Heuristic Scoring Compares to LLM Semantic Self-Healing Under the Hood

What problem self-healing locators solve, and one claim to debunk first

The most fragile part of Web UI automation testing isn't the assertion logic — it's element location. id, class, and the DOM hierarchy can all break the moment a redesign ships. When the script errors out, it's not the business logic that failed; it's "the selector for this button can't be found." **Self-Healing Locators** exist to solve exactly this: when a locator fails, the system tries to automatically find a new locator for what "looks like the same element," instead of letting the test go red and waiting for a human to fix the selector.

The oldest and most-starred open-source project in this space is Healenium (healenium/healenium-web, 201 stars, last commit on 2026-09-07, still actively maintained). Its README says: "leverages machine learning to allow Selenium tests to self-heal." The first thing this article does is read the source code to verify that claim — the conclusion: the claim isn't wrong in a marketing sense, but it's misleading in a technical sense. Healenium's core self-healing mechanism isn't machine learning; it's a weighted-scoring heuristic formula, where the weights are hardcoded floating-point constants. Training data and model inference — the basic ingredients of machine learning — simply don't exist in this formula.

Let's be clear about the three questions this article answers: How exactly does Healenium's "so-called machine learning" heuristic scoring compute its score, and where's the ceiling? What form does the AI capability added in 2026 actually take, and why is it locked behind a paywall right in the code? And by contrast, how does the genuinely LLM-based open-source semantic self-healing project — MarketSquare/robotframework-selfhealing-agents — architect "relocating by what the element does rather than where it is"?

Healenium's scoring formula: tree similarity + Levenshtein distance, with zero machine learning involved

Healenium's self-healing flow has two layers: SelfHealingEngine (the healenium-web main repo) intercepts locator failures at test runtime, parses the current page's DOM into a Node tree (via JsoupHTMLParser), then hands it off to the separately published tree-comparing (Maven coordinates com.epam.healenium:tree-comparing, with the version pinned to pom.xml in healenium-web's 0.4.14) to do tree-matching scoring.

The core matching logic lives in PathFinder.findScoresToNodes(): it first uses LCSPathDistance (longest common subsequence) to compare "the failed element's historical path" against "the paths of all leaf nodes on the current page," finds the structurally most similar candidate path, then calls HeuristicNodeDistance.distance() to score each node along that candidate path. This scoring function was verified by directly decompiling the bytecode (the tree-comparing-0.4.14.jar on Maven Central — the floating-point constants in the constant pool match the source exactly). The formula looks like this:

private static final double POINTS_FOR_TAG = 100.0D;
private static final double POINTS_FOR_ID = 50.0D;
private static final double POINTS_FOR_CLASS = 40.0D;
private static final double POINTS_FOR_VALUE = 30.0D;
private static final double POINTS_FOR_OTHER_ATTRIBUTE = 30.0D;
// maximumScore = 350.0

double score = LCSDistance / curPathHeight * 100.0;   // 路径结构相似度部分
if (tag相同) score += 100.0;
if (id存在) score += 50.0 * levenshteinScore(id1, id2, 0.3);
score += 30.0 * levenshteinScore(innerText1, innerText2, 0.3);
score += classNamesIntersect比例 * 40.0;
score += otherAttributes逐项Levenshtein均值 * 30.0;
return score / maximumScore;   // 归一化到 [0, 1]

Internally, calculateLevenshteinScore() uses Apache Commons Text's LevenshteinDistance to convert the edit distance between two strings into a similarity value in [0,1]. The threshold parameter (0.3 or 0.75) determines how much edit distance is still considered similar. There's no "learning" component anywhere in this formula — the weights (100, 50, 40, 30) are hardcoded constants, not parameters fitted from data; the similarity algorithm is string edit distance, not word vectors or semantic embeddings. If an element's id changes from login-btn to signin-btn, the Levenshtein distance is small, the score is high, and it's judged to be "the same element." But if it changes to an submit-auth that's semantically identical yet completely different as a string, the edit distance maxes out and the score plummets — even though to a human it's obviously "the same login button." This is exactly the ceiling of heuristic scoring: it measures surface similarity of strings and structure, not the semantic similarity of what the element does — which is why the "LLM semantic matching self-healing: relocate by 'what this element does' rather than 'where this element is'" mentioned in the plan is a real technical gap, not a need invented by marketing copy.

The final score is threshold-filtered by HealingService using the configured score-cap (application.conf defaults to .6, the test config uses .55). Candidate nodes above the threshold are sorted by score descending, converted one by one into CSS selectors or XPath, and validated with driver.findElements() to see whether they uniquely locate an element — the first one that validates is the self-healing result. This is a fully deterministic rules engine: fast, explainable, and requiring no external API calls. But its boundaries are equally clear: as long as the change happens to fall outside the comparable dimensions of "tag, id, class, text, attributes" — say, the whole component is reimplemented but keeps the same semantics — this formula has no way to judge it.

The AI interface added in 2026: a path blocked directly by a paywall

In RestClient.java there's an independent method that doesn't belong to the scoring logic above:

public String getXpathSelector(Node node, String sessionId) {
    HttpRequest request = new HttpRequest(HttpMethod.POST, "/selectors/xpath");
    ...
    HttpResponse response = aiServiceExecute(request);
    if (HTTP_NOT_FOUND == response.getStatus()) {
        throw new RuntimeException("[Get Xpath Selector] Compatibility error. You must have a paid hlm-ai service.");
    }
    ...
}

This method POSTs DOM node info to a separate aiServiceUrl (the config item is hlm.ai.url, which defaults to application.conf pointing at http://localhost:6565), fully decoupled from the scoring engine — HealingService.createXPathFromElement() only calls it when useXPath(engine) evaluates to true (i.e., selector-type is configured as xpath), as an alternative path beyond CSS selector construction. This code itself confirms two things: first, the Healenium team itself acknowledges that pure rule-based scoring isn't enough, so they opened a dedicated channel to "let an AI service generate XPath"; second, that channel is explicitly blocked behind a paywall — receiving 404 throws an exception directly, with error text blunt enough to need no embellishment: "You must have a paid hlm-ai service." The hlm-ai service itself is a closed-source commercial component, outside the scope of the open-source repo, so what model it uses and how it does semantic understanding can't be verified externally at all.

This detail is more valuable than the judgment "whether a high-star tool's AI marketing is true": it shows that even the oldest and longest-maintained open-source project in this space believes pure rule-based scoring can no longer hold up and needs semantic understanding — but that capability was chosen to be closed-source and commercialized rather than built openly. This isn't the same situation as the pattern you'll see later in 11 (high-star projects whose "intelligence" marketing often wraps non-LLM mechanisms): Healenium isn't false advertising — it honestly admits its limitations, then turns the patch into a paid feature.

Real LLM self-healing: MarketSquare/robotframework-selfhealing-agents' orchestrator-locator dual-agent architecture

MarketSquare/robotframework-selfhealing-agents (27 stars, Apache-2, created 2025-4, last commit 2026-2, MarketSquare being the official organization under the Robot Framework ecosystem) is a rare open-source implementation in this space where you can directly confirm from the source that it calls a real LLM API. It's a Listener plugin for Robot Framework that handles locator failures with a two-layer agent architecture:

Layer one: OrchestratorAgent, built on PydanticAI, whose job is to determine "is this failure a locator problem?" Its judgment logic is itself delegated to the locator_agent.is_failed_locator_error() downstream in the LLM call chain — for example, in the Selenium case, checking whether the error text matches patterns like "with locator" ... "not found". If it's not a locator failure (say, an assertion failure or network timeout), it returns NoHealingNeededResponse directly, without wasting tokens on the repair flow. Only after it's judged a locator failure is the request routed to a pydantic-ai Agent, registering "repair the locator" as a callable tool via output_type=[self._get_healed_locators] and running a real LLM tool-call loop.

Layer two: BaseLocatorAgent (and its SeleniumLocatorAgent/BrowserLocatorAgent subclasses) is the part that actually does semantic location. It supports two working modes, switched by the config item use_llm_for_locator_generation (default True):

• Pure LLM generation mode: it stuffs the DOM tree at failure time, the error message, the failed locator string, and the list of locators already tried and still failing into the prompt, and lets the model directly output 3 new locator candidates. The system prompt is blunt: "Using the elements in the DOM at failure time, suggest 3 new locators... Make sure you do not suggest a locator that is on that list." — this is pure language-model semantic reasoning, with no rule-based scoring involved.
• DOM tool generation + LLM selection mode: it first uses deterministic code (_dom_utility.get_locator_proposals()) to enumerate candidate locators based on DOM structure, then has a separate selection_agent pick the "most suitable" one from the candidate list. This path is a combination of "deterministic generation + LLM judgment." Compared with pure LLM generation mode, it lowers the risk of hallucinating invalid selectors, but it also leaves less room for semantic understanding — the LLM can only choose among the already-enumerated candidates, and can't produce an answer outside that candidate set.

The trade-off between these two modes happens to be a concrete answer to the planning question of "the balance between self-healing success rate and false-positive rate": the pure LLM generation mode has broader coverage (the model can come up with locator strategies that the candidate enumeration missed), but it's more prone to hallucinated output that is syntactically valid yet selects no element; the DOM tool generation mode is more conservative and more reliable, but its ceiling is capped by "what deterministic enumeration can find." The project chose to expose this switch to users rather than deciding for them, and that design itself acknowledges that both paths have costs—neither one can simultaneously achieve "high coverage" and "low false-positive rate."

There's one more engineering detail in the response validation layer worth noting: generation_agent.output_validator runs another deterministic filtering pass after receiving the model output—_sort_locators() uses is_locator_unique() to rank candidates that "uniquely locate an element" first, and _filter_clickable_locators() targets failure scenarios involving keywords like click/tap/select, specifically filtering out elements that are actually clickable. If no candidate passes validation in the end, the code throws ModelRetry, triggering pydantic-ai's built-in retry mechanism instead of directly returning a possibly wrong locator. This differs from a simple "whatever the model says goes" invocation—the LLM's semantic judgment result still has to pass through a layer of deterministic code validation before being trusted, which splits "semantic understanding" and "structural correctness" into two independent responsibilities rather than expecting a single model call to get both right at once.

The configuration layer (SelfhealingAgents/utils/cfg.py) confirms this is a genuine multi-provider architecture: orchestrator_agent_provider/locator_agent_provider both support openai/azure, the default model is gpt-4o-mini, and there are separate request_limit (default 5) and total_tokens_limit (default 6000) enforcing hard cost constraints—these hard constraints are themselves a direct response to the concern of "will LLM self-healing run away and burn tokens": each self-healing request has a clear token consumption cap, and exceeding it terminates immediately rather than retrying indefinitely.

The balance between self-healing success rate and false-positive rate—where does it get stuck?

Putting the mechanisms of the two projects side by side, the planning question of "when is self-healing masking a real UI regression bug" has a concrete answer: whether it's heuristic scoring or LLM semantic judgment, the self-healing mechanism itself cannot distinguish "this is a harmless implementation detail change" from "this is a genuine functional UI regression"—what they judge is "does something on the new page look like the old element," not "does this change match product expectations." Healenium's scoring formula might give a similarly high score to a pure id rename and to a case where a button was accidentally deleted and then a visually similar but functionally different new button was added; LLM semantic judgment, when faced with "a new button with similar text but completely different logic added to the page," can likewise be misled by surface-level semantic similarity.

This means the reasonable boundary of the self-healing mechanism is not "improve judgment accuracy to near 100%," but rather treating self-healing results as a signal that needs a paper trail and human review, not an automatic fix that can be fully trusted and directly turns tests green. robotframework-selfhealing-agents's reporting system (reports/report_generator.py and its series of report_types) specifically generates healed_files_report/diff_files_report—this design itself acknowledges that "self-healing succeeded" does not equal "there's no problem," but instead records every self-healing event as a change that needs review, for humans to judge whether a real UI regression is hiding behind the fix. This is the same idea as Healenium's backlight-healing configuration option (default true, which highlights in the test report which elements were recovered by self-healing)—two projects with completely different technical approaches made the same engineering choice on the point that "self-healing results must be traceable and reviewable," and this says more about where the consensus boundary in this field lies than any single scoring formula or prompt design.

Summary

1. The "leverages machine learning" claim in the Healenium README is marketing exaggeration—the core self-healing mechanism (HeuristicNodeDistance.distance()) is a weighted scoring formula with hardcoded weights (TAG=100, ID=50, CLASS=40, VALUE=30), and the similarity calculation uses Levenshtein edit distance, which has nothing to do with the training/inference paradigm of machine learning; the authenticity of the source code has been verified by decompiling the bytecode constant pool of tree-comparing-0.4.14.jar on Maven Central.
2. The RestClient.getXpathSelector() added in Healenium 2026 is an independent AI-generated XPath channel, and the code directly throws a "You must have a paid hlm-ai service" exception for 404 responses—even the most established open-source project in this field acknowledges that pure rule-based scoring can't keep up with the demands of semantic understanding, but it chose to make this capability a closed-source paid service rather than building it together in the open.
3. MarketSquare/robotframework-selfhealing-agents is a genuinely LLM-based open-source self-healing implementation, using a two-layer architecture of an orchestration agent (judging whether self-healing is needed) plus a locator agent (generating/selecting new locators), supporting two modes—"pure LLM generation" and "DOM enumeration + LLM selection"—which make different trade-offs between coverage and false-positive rate, and the project leaves this choice to the user rather than baking in a default answer.
4. The project's LLM output is not directly trusted—the output_validator layer uses deterministic code to validate the uniqueness and clickability of candidate locators, and when validation fails it triggers a ModelRetry retry rather than directly returning a possibly wrong result, splitting semantic judgment and structural correctness verification into two independent responsibilities.
5. Whether it's heuristic scoring or LLM semantic judgment, the self-healing mechanism itself cannot distinguish "harmless implementation adjustment" from "genuine functional regression"—two projects with completely different technical approaches respectively use backlight-healing highlighting and structured healed/diff reports, choosing the same response: treating self-healing results as a signal that needs a paper trail and human review, not a fully trustworthy automatic fix.

Welcome to PrimeSkills — a curated marketplace of AI Agents and skills, all validated against real enterprise-grade workflows. No hype, just what actually works.

For more practical knowledge and interesting products, welcome to visit my personal homepage