Workflow Performance Benchmarking¶

This article documents the results of structured load testing carried out on IFS Cloud BPA Workflows. The primary objective was to benchmark IFS Cloud BPA Workflows and understand how simple and complex loop-heavy Workflows behave under concurrent user load, using loop cardinality, Workflows per execution and resource consumption as the key metrics.

Two representative Workflow patterns were tested: a lightweight synchronous Workflow and a complex asynchronous loop Workflow. Load was applied at 1 to 300 concurrent users. During this testing, the asynchronous loop Workflow showed progressive reliability issues as Loop Cardinality and concurrency increased. That observation led to a follow-up test in which the heap setting was adjusted to see whether it changed the outcome, and by how much — this is documented as a separate performance improvement test outcome further down in this article, alongside the original benchmark results.

Test Environment¶

All testing was conducted on a dedicated IFS Cloud environment. No other business workloads, integrations, reports or user activity ran concurrently during any test window.

Tested Workflows¶

Two BPA Workflows were purpose-built and pushed to their limits. Both were deliberately left un-optimized — no filters, no parallel execution, no payload trimming were applied. This establishes a conservative baseline; real optimized Workflows can be expected to perform the same or better.

Simple Synchronous Workflow¶

A lightweight, synchronous Workflow used to establish a baseline for request/response-style Workflows that complete within the triggering request itself.

​ Figure 1 — Simple Workflow design: synchronous execution, no loops

Bulk Asynchronous Loop Workflow¶

A background, asynchronous Workflow built around sequential loops, used to push a worst-case, unoptimized loop-heavy design to its limits.

​ Figure 2 — Bulk Data Workflow design: asynchronous background execution with sequential loops

Note on attribute count: The Simple Workflow tests a 184-attribute entity — a deliberately demanding case for a synchronous Workflow. The Bulk Workflow tests an 18-attribute entity. Loop timings measured here reflect a lightweight entity; loops over wider entities will cost proportionally more per iteration.

Load Test Parameters¶

The following abbreviations are used throughout this document:

Simple Workflow Results¶

This section covers the results of the Simple Workflow tests, ran across LP1 to LP300 and a 1-hour endurance run.

Simple Workflow Trigger Response Times

The table below shows trigger response times for each LP, plus the 1-hour endurance run.

The single error at LP300 (0.33%) did not reproduce when the ramp-up was extended to 900 seconds (0.00% errors), confirming it was a brief arrival-burst artifact rather than a capacity limit. The endurance-test errors were isolated HTTP 500 error codes in a single 5-minute window; all Workflow instances completed successfully per Workflow engine history.

Simple Workflow Database Resource Usage

The table below shows database CPU, memory and disk write activity observed during the Simple Workflow endurance and burst runs.

Database resource usage stayed low throughout, confirming the database was not a limiting factor for the Simple Workflow pattern.

Simple Workflow Throughput

The table below shows measured throughput for each LP and the endurance run.

Note: Execution is synchronous, so every successful request is a completed Workflow. In the paced burst runs, measured throughput matched the configured arrival rate exactly with no queuing. The endurance throughput reflects the test-plan pacing, not a platform ceiling (database CPU stayed at 2–5% throughout).

Bulk Workflow Results¶

This section covers the results of the Bulk Data Workflow tests, ran across LP1–LP300 and loop cardinalities from 1 to 2,000, to benchmark performance and establish the point at which the Workflow execution starts to fail.

Outcomes by Load Profile and Loop Cardinality

The table below shows the outcome of representative LP and LC combinations. As LC increases, Workflow instances increasingly needed async retries to complete, and at the highest Loop Cardinality tested, the Workflow failed outright and the OData PODs became unresponsive.

Bulk Workflow Database Resource Usage

The table below shows database CPU, memory and disk write activity during the heaviest run, where the Workflow itself was failing.

DB CPU stayed below 25% and memory remained relatively steady even while the Workflow itself was failing — an early signal that the database was not the bottleneck behind these failures.

Root Cause Analysis¶

This subsection explains what was found when the failures observed above were investigated. The first area reviewed was the OData POD itself, and its JVM heap setting (MaxRAMPercentage) was found to be at its default value of 95%. The OData POD runs a JVM inside a Kubernetes container with a hard 4 GB memory limit (cgroup limit). MaxRAMPercentage controls what fraction of that 4 GB the JVM may allocate as heap. The remainder must cover all JVM internals operating outside the heap:

At the default 95% setting, the heap cap is ~3.8 GB, leaving only ~200 MB for everything else — insufficient under sustained loop-Workflow load.

When the total footprint exceeds 4 GB the kernel OOMKills the JVM process. Kubernetes detects the container exit and restarts it. This is visible as unexpected OData POD restarts in the Kubernetes events and in the BPA_POD_HISTORY_TAB table. This pointed directly at the JVM heap setting as the next thing to test, rather than the database or the Workflow design itself. That follow-up test is documented in Heap Setting Change — Performance Improvement Test below.

Heap Setting Change — Performance Improvement Test¶

Both the Simple and Bulk Workflow test suites above were run using the OData POD's default configuration. As explained in Root Cause Analysis, this meant the JVM heap setting (MaxRAMPercentage) was at its default value of 95%. As the first step to test whether this had any effect, MaxRAMPercentage was changed to 70% — keeping every other environment parameter and configuration unchanged — and the same tests were re-run. This section presents that test outcome for both Workflow patterns, followed by the resulting performance recommendation.

Simple Workflow — Heap 70% Comparison¶

To confirm the heap setting change did not negatively affect the lightweight, synchronous Workflow pattern, the Simple Workflow 1-hour endurance test was repeated at heap 70%.

Response times, database resource usage and throughput were consistent with the heap 95% endurance run in Simple Workflow Results above — confirming that the heap setting has no measurable effect on this simple, synchronous Workflow pattern.

Bulk Workflow — Heap 70% Comparison¶

MaxRAMPercentage was lowered from the default 95% to 70% and the Bulk Workflow test suite was re-run in full, across the same LPs and LCs, to see whether reliability improved. An intermediate value of 80% was also tested for comparison. This section presents that comparison as a test outcome — it is not a starting configuration, but a change identified through this testing.

Key finding: Setting MaxRAMPercentage to 70% on the OData POD delivered 100% Workflow success at every tested load level — including 300 concurrent users each processing 2,000 records per execution. The default value of 95% caused progressive degradation and, at higher loads, complete Workflow failure and POD unresponsiveness, as shown earlier.

The chart below plots median Workflow duration against Loop Cardinality at heap 70%, across the four LPs.

​ Figure 3 — Median Workflow Duration vs Loop Cardinality: LP1, LP50, LP100 and LP300 (Heap 70%)

Bulk Workflow Duration at Heap 70%

The table below shows median Workflow duration (seconds) across LPs and LCs, at heap 70%. Every run in this table completed 100% of Workflow instances with no errors and no async retries.

Key insight: at 300 concurrent users with proportional 5 seconds per user pacing, completion times track the 100-trigger curve — and at smaller loop cardinalities finish even faster. Arrival spacing is the dominant factor controlling queueing. At LP300 & ramp 15,000 s, the median for LC 2,000 dropped from 1,353 seconds to 449 seconds — a 3× improvement achieved purely by spreading trigger arrivals.

Heap Setting Comparison at Peak Load

The identical heaviest test was run three times, changing only MaxRAMPercentage:

Note: The database was never the bottleneck — DB CPU stayed below 23% and memory was stable even in the failing runs. Failures at 95% were application-tier, not database exhaustion.

The chart below compares heap 70% and heap 95% at LP300 with a 1,500 s ramp as Loop Cardinality increases. Duration stays clean at 70%; at 95% retries grow with Loop Cardinality and the run fails at LC 2,000.

Figure 4 — LP300 & Ramp 1,500 s: Heap 70% vs Heap 95% — Duration and Outcome by Loop Cardinality. The green line shows clean completions at 70%; the orange line shows increasing retries at 95% as loop cardinality grows; the red X marks complete failure at LC 2,000 across every LP.

The chart below shows the heaviest test — 300 concurrent users, 2,000-record loops, 1,500 s ramp — run three times, changing only MaxRAMPercentage. Heap 70% completed every instance; 80% completed only with retries and POD scaling; 95% failed almost entirely.

Figure 5 — Workflow Outcomes by MaxRAMPercentage at LP300, LC 2,000, Ramp 1,500 s: the identical heaviest test run three times, changing only MaxRAMPercentage.

Heap 70% versus 95% Outcome Comparison

The table below lists matching load and loop combinations at heap 70% and heap 95%, side by side. Behaviour is identical at smaller loops; retries appear first at 95%, then complete failure at LC 2,000 regardless of concurrent users or ramp.

Bulk Workflow Database Resource Usage at Heap 70%

The table below shows database CPU, memory and disk write volume for representative Bulk Workflow runs at heap 70%. DB CPU stayed below 23% and memory remained steady — consistent with the default-heap run shown earlier, confirming the database was never the bottleneck at either setting.

Why Heap 80% is not a safe middle ground: Load testing showed that 80% still causes async retries and POD scaling under the same Workloads that run cleanly at 70%. Retries consume extra capacity and mask the approaching limit.

Performance Recommendation¶

The benchmark results demonstrated that configuring MaxRAMPercentage=70.0 improved reliability and execution performance for the tested loop-heavy asynchronous Workflow workload. Under the same workload, the default setting of MaxRAMPercentage=95.0 resulted in increasing async retries, OData POD instability, and ultimately Workflow failures at higher load levels.

This recommendation is specific to the environment, Workflow design, and workload characteristics used in this benchmark. Customers should validate the configuration against their own workload requirements before applying it to production environments.

IFS Cloud Hosted Customers¶

Customers using IFS Cloud Hosted environments cannot modify OData POD JVM settings directly.

If evaluation of this recommendation is required, contact IFS Cloud Support to determine whether the configuration change is appropriate for the target environment and workload.

IFS Remote Customers¶

Remote customers can evaluate and apply this configuration change on the ifsapp-odata deployment. The ODP_JAVA_OPTS environment variable controls the JVM startup parameters used by the OData POD.

Recommended Configuration for the Tested Workload¶

The table below summarizes the configuration change required to apply this recommendation.

Recommended ODP_JAVA_OPTS Value

Update only the MaxRAMPercentage parameter and retain all other JVM startup options unchanged.

-XX:MaxRAMPercentage=70.0 -XX:-OmitStackTraceInFastThrow -Dodp.projection.cache.limit=500 

Steps to Apply the Heap Setting

The table below lists the steps to apply and verify the heap setting change.

Rollback Considerations

If unexpected application behavior is observed after applying the configuration change, restore the previous MaxRAMPercentage value and restart the affected OData PODs.

Configuration changes should be introduced through normal change management processes and validated in a non-production environment whenever possible before adoption in production systems.

Monitor Workflow Health

After applying the heap setting change, use the checks below to confirm the environment is behaving as expected and to catch any early warning signs of heap pressure.

Check Async Workflow Retries and Failures

Run the following against the IFS database to check for retried or failed async executions for your tested Workflows. On a correctly configured system (if no Workflow failures has occurred), all rows should have RETRY_COUNT = 1:

SELECT BPA_KEY, STATUS, RETRY_COUNT, ROWVERSION 
FROM   BPA_ASYNC_SYS_AUDIT_TAB 
WHERE  RETRY_COUNT > 1 OR STATUS = 'FAILED' 
ORDER  BY ROWVERSION DESC; 

Check OData POD Restarts

Multiple distinct PODs registering in a short window indicates unexpected scaling or restarts:

SELECT POD_NAME, POD_STARTUP_TIME, LAST_UPDATED 
FROM   BPA_POD_HISTORY_TAB 
ORDER  BY POD_STARTUP_TIME DESC; 

Workflow Health Indicators

The table below summarizes the indicators to monitor and the action to take if each one is NOT met.

Early-warning signal: Retries appear before failures. At heap 95%: 11 retried at LC 500 → 28 retried at LC 750 → 139 retried at LC 1,000 → complete failure at LC 2,000. Monitoring RETRY_COUNT > 1 gives advance warning of approaching collapse. On a correctly configured system, the expected retry count is 1 (complete on 1st attempt) for your tested Workflows.

Benchmark Summary and Conclusions¶

The benchmark shows that IFS Cloud BPA Workflows can execute reliably at substantial load when Workflow complexity, trigger arrival rate, OData POD memory configuration, and environment capacity are aligned. The figures below describe the tested operating envelope of the dedicated Large (L) environment used for this benchmark. They are sizing examples, not platform-wide limits, guarantees, or service-level commitments.

Simple Synchronous Workflow Summary¶

The Simple Workflow represented a lightweight request/response pattern without loops. Each execution performed three database interactions against a wide 184-attribute entity and completed within the triggering request.

Tested Capacity and Results

Simple Workflow Conclusion

• Up to 300 concurrent simple synchronous Workflows were tested successfully when ramped over 900 seconds.
• The endurance tests sustained approximately 53–55 completed Workflows per minute.
• Median response time remained close to 1.1 seconds at the tested load levels.
• Lowering JVM param MaxRAMPercentage from 95% to 70% had no measurable effect on this Workflow pattern.
• Database utilization remained low, indicating that the database was not a limiting component for the tested Simple Workflow.

For a comparable Large environment, 300 concurrent executions and approximately 53–55 executions per minute can be used as tested reference points for a simple Workflow of similar complexity. The per-minute result reflects the configured test pacing and must not be interpreted as the maximum platform capacity.

Bulk Asynchronous Workflow Summary¶

The Bulk Workflow represented a loop-heavy asynchronous pattern. Each instance created and deleted records using sequential loops. At Loop Cardinality 2,000, one Workflow performed approximately 4,002 database interactions.

Tested Capacity and Results

Bulk Workflow Conclusion

• At MaxRAMPercentage=70.0, the environment safely completed 300 concurrent asynchronous Workflows, each with a Loop Cardinality of 2,000.
• The successful peak test represented 600,000 create iterations, 600,000 delete iterations, and approximately 1.2 million database interactions across the run.
• All 300 instances completed on the first attempt, with zero retries, zero failures, and no POD restarts.
• Increasing the ramp from 1,500 to 15,000 seconds reduced median duration from 1,353.1 seconds to 449.3 seconds, demonstrating the effect of trigger pacing and queueing.
• The identical workload was not safe at the default 95% heap setting, where 292 of 300 Workflow instances failed.
• Retries appeared before complete failure at heap 95%: 11 of 300 instances retried at Loop Cardinality 500, 28 at 750, and 139 at 1,000. Therefore, RETRY_COUNT > 1 should be treated as an early capacity warning.
• Database CPU remained below 23% during the heavy tests. The observed failures were associated with application-tier memory pressure rather than database exhaustion.

For a comparable Large environment with MaxRAMPercentage=70.0, 300 concurrent asynchronous Workflows with Loop Cardinality 2,000 can be used as the tested upper reference point from this benchmark. This is not a general safe limit for all asynchronous Workflows because entity width, loop design, payload size, nested loops, trigger arrival rate, environment size, and competing workloads can materially change capacity.

Overall Interpretation¶

• Use the Simple Workflow figures when the Workflow contains a small number of steps, no loops, and completes within the triggering request.
• Use the Bulk Workflow figures when the Workflow runs asynchronously and performs high-volume sequential loop processing.
• Do not combine the two figures into a single daily Workflow limit. The execution cost of the two patterns is substantially different.
• Before production rollout, repeat the benchmark with representative business data and competing workloads. Monitor async retries, failed executions, OData POD restarts, Workflow duration, and infrastructure utilization.

Frequently Asked Questions¶