Project

General

Profile

Bug #2196

Deadlock in Make

Added by Henning Blohm 6 days ago. Updated 6 days ago.

Status:
In Progress
Priority:
High
Assignee:
Henning Blohm
Category:
z2-core
Target version:
2.10.2
Start date:
12.06.2025
Due date:
% Done:

0%

Estimated time:
origin:

Description

As found in testing:


Found one Java-level deadlock:
=============================
"DEFAULT_Worker-11":
  waiting to lock monitor 0x00007f3f7403f200 (object 0x00000000d3e50b90, a com.zfabrik.impl.components.java.JavaComponentImpl),
  which is held by "pool-5-thread-3" 
"pool-5-thread-3":
  waiting to lock monitor 0x00007f3f94a96300 (object 0x00000000c06080e0, a com.zfabrik.impl.components.java.ComponentsBuilder),
  which is held by "DEFAULT_Worker-11" 

Java stack information for the threads listed above:
===================================================
"DEFAULT_Worker-11":
        at com.zfabrik.impl.components.java.JavaComponentImpl.as(JavaComponentImpl.java:87)
        - waiting to lock <0x00000000d3e50b90> (a com.zfabrik.impl.components.java.JavaComponentImpl)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.NamespaceImpl.lambda$resolve$0(NamespaceImpl.java:229)
        at com.zfabrik.impl.resources.NamespaceImpl$$Lambda$100/0x000000010017bc40.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.NamespaceImpl.resolve(NamespaceImpl.java:201)
        at com.zfabrik.impl.resources.NamespaceImpl.lookup(NamespaceImpl.java:245)
        at com.zfabrik.components.IComponentsLookup$initializer$1.lookup(IComponentsLookup.java:72)
        at com.zfabrik.impl.components.java.ComponentsBuilder._makePart(ComponentsBuilder.java:535)
        at com.zfabrik.impl.components.java.ComponentsBuilder.make(ComponentsBuilder.java:352)
        - locked <0x00000000c06080e0> (a com.zfabrik.impl.components.java.ComponentsBuilder)
        at com.zfabrik.impl.components.java.JavaComponentImpl.lambda$as$0(JavaComponentImpl.java:154)
        at com.zfabrik.impl.components.java.JavaComponentImpl$$Lambda$105/0x000000010017e840.run(Unknown Source)
        at java.security.AccessController.doPrivileged(java.base@11.0.21/Native Method)
        at com.zfabrik.impl.components.java.JavaComponentImpl.as(JavaComponentImpl.java:93)
        - locked <0x00000000d3e511c8> (a com.zfabrik.impl.components.java.JavaComponentImpl)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:100)
        at com.zfabrik.impl.springframework.ContextResource._load(ContextResource.java:121)
        at com.zfabrik.impl.springframework.ContextResource.as(ContextResource.java:77)
        - locked <0x00000000d3e51298> (a com.zfabrik.impl.springframework.ContextResource)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:100)
        at com.zfabrik.impl.springframework.BeanResource._load(BeanResource.java:101)
        at com.zfabrik.impl.springframework.BeanResource.as(BeanResource.java:74)
        - locked <0x00000000d3e51350> (a com.zfabrik.impl.springframework.BeanResource)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.NamespaceImpl.lambda$resolve$0(NamespaceImpl.java:229)
        at com.zfabrik.impl.resources.NamespaceImpl$$Lambda$100/0x000000010017bc40.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.NamespaceImpl.resolve(NamespaceImpl.java:201)
        at com.zfabrik.impl.resources.NamespaceImpl.lookup(NamespaceImpl.java:245)
        at com.zfabrik.components.IComponentsLookup$initializer$1.lookup(IComponentsLookup.java:72)
        at com.tracekey.mytk.util.extensions.ExtensionPoints.lambda$0(ExtensionPoints.java:64)
        at com.tracekey.mytk.util.extensions.ExtensionPoints$$Lambda$700/0x0000000100c8ac40.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(java.base@11.0.21/ReferencePipeline.java:195)
        at java.util.Iterator.forEachRemaining(java.base@11.0.21/Iterator.java:133)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(java.base@11.0.21/Spliterators.java:1801)
        at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.21/AbstractPipeline.java:484)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.21/AbstractPipeline.java:474)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@11.0.21/ReduceOps.java:913)
        at java.util.stream.AbstractPipeline.evaluate(java.base@11.0.21/AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(java.base@11.0.21/ReferencePipeline.java:578)
        at com.tracekey.mytk.util.extensions.ExtensionPoints.getAllAs(ExtensionPoints.java:66)
        at com.tracekey.mytk.util.imports.impl.service.ImportServiceImpl.getExplorers(ImportServiceImpl.java:146)
        at com.tracekey.mytk.util.imports.impl.service.ImportServiceImpl.checkAndHandleImportsByUseCases(ImportServiceImpl.java:67)
        at com.tracekey.connectors.extension.impl.cron.ConnectorCronJob.execute(ConnectorCronJob.java:45)
        at com.tracekey.mytk.util.scheduling.impl.service.JobWrapper.lambda$2(JobWrapper.java:63)
        at com.tracekey.mytk.util.scheduling.impl.service.JobWrapper$$Lambda$688/0x0000000100c8dc40.call(Unknown Source)
        at com.tracekey.mytk.util.api.tenant.TenantScope.call(TenantScope.java:125)
        at com.tracekey.mytk.util.scheduling.impl.service.JobWrapper.lambda$1(JobWrapper.java:62)
        at com.tracekey.mytk.util.scheduling.impl.service.JobWrapper$$Lambda$687/0x0000000100c8d840.call(Unknown Source)
        at com.zfabrik.util.threading.ThreadUtil.lambda$cleanContextExceptionExecute$0(ThreadUtil.java:55)
        at com.zfabrik.util.threading.ThreadUtil$$Lambda$116/0x000000010018ec40.run(Unknown Source)
        at java.security.AccessController.doPrivileged(java.base@11.0.21/Native Method)
        at com.zfabrik.util.threading.ThreadUtil.cleanContextExceptionExecute(ThreadUtil.java:55)
        at com.tracekey.mytk.util.scheduling.impl.service.JobWrapper.lambda$0(JobWrapper.java:52)
        at com.tracekey.mytk.util.scheduling.impl.service.JobWrapper$$Lambda$683/0x0000000100c8e840.run(Unknown Source)
        at com.zfabrik.impl.work.ThreadPoolImpl$$Lambda$684/0x0000000100c8ec40.run(Unknown Source)
        at com.zfabrik.impl.work.ThreadPoolImpl.lambda$runAs$10(ThreadPoolImpl.java:427)
        at com.zfabrik.impl.work.ThreadPoolImpl$$Lambda$685/0x0000000100c8e040.get(Unknown Source)
        at com.zfabrik.work.WorkUnit.supply(WorkUnit.java:378)
        at com.zfabrik.impl.work.ThreadPoolImpl.getAs(ThreadPoolImpl.java:383)
        at com.zfabrik.impl.work.ThreadPoolImpl.runAs(ThreadPoolImpl.java:426)
        at com.zfabrik.impl.work.ThreadPoolImpl.executeAs(ThreadPoolImpl.java:361)
        at com.tracekey.mytk.util.scheduling.impl.service.JobWrapper.execute(JobWrapper.java:43)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c46011e8> (a java.lang.Object)
"pool-5-thread-3":
        at com.zfabrik.impl.components.java.ComponentsBuilder.make(ComponentsBuilder.java:140)
        - waiting to lock <0x00000000c06080e0> (a com.zfabrik.impl.components.java.ComponentsBuilder)
        at com.zfabrik.impl.components.java.JavaComponentImpl.lambda$as$0(JavaComponentImpl.java:154)
        at com.zfabrik.impl.components.java.JavaComponentImpl$$Lambda$105/0x000000010017e840.run(Unknown Source)
        at java.security.AccessController.doPrivileged(java.base@11.0.21/Native Method)
        at com.zfabrik.impl.components.java.JavaComponentImpl.as(JavaComponentImpl.java:93)
        - locked <0x00000000d3e50b90> (a com.zfabrik.impl.components.java.JavaComponentImpl)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:100)
        at com.zfabrik.impl.components.java.JavaComponentImpl.addDependencies(JavaComponentImpl.java:192)
        at com.zfabrik.impl.components.java.JavaComponentImpl.lambda$as$0(JavaComponentImpl.java:138)
        at com.zfabrik.impl.components.java.JavaComponentImpl$$Lambda$105/0x000000010017e840.run(Unknown Source)
        at java.security.AccessController.doPrivileged(java.base@11.0.21/Native Method)
        at com.zfabrik.impl.components.java.JavaComponentImpl.as(JavaComponentImpl.java:93)
        - locked <0x00000000d3ea1ee8> (a com.zfabrik.impl.components.java.JavaComponentImpl)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:100)
        at com.zfabrik.impl.components.java.JavaComponentImpl.addDependencies(JavaComponentImpl.java:192)
        at com.zfabrik.impl.components.java.JavaComponentImpl.lambda$as$0(JavaComponentImpl.java:138)
        at com.zfabrik.impl.components.java.JavaComponentImpl$$Lambda$105/0x000000010017e840.run(Unknown Source)
        at java.security.AccessController.doPrivileged(java.base@11.0.21/Native Method)
        at com.zfabrik.impl.components.java.JavaComponentImpl.as(JavaComponentImpl.java:93)
        - locked <0x00000000d3ea2070> (a com.zfabrik.impl.components.java.JavaComponentImpl)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:100)
        at com.zfabrik.impl.springframework.ContextResource._load(ContextResource.java:121)
        at com.zfabrik.impl.springframework.ContextResource.as(ContextResource.java:77)
        - locked <0x00000000d3ea21d0> (a com.zfabrik.impl.springframework.ContextResource)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:100)
        at com.zfabrik.impl.springframework.BeanResource._load(BeanResource.java:101)
        at com.zfabrik.impl.springframework.BeanResource.as(BeanResource.java:74)
        - locked <0x00000000d3ef2088> (a com.zfabrik.impl.springframework.BeanResource)
        at com.zfabrik.impl.components.ComponentResourceWrapper.as(ComponentResourceWrapper.java:91)
        at com.zfabrik.resources.provider.Resource.as(Resource.java:153)
        at com.zfabrik.impl.resources.ResourceHandleImpl.lambda$as$0(ResourceHandleImpl.java:89)
        at com.zfabrik.impl.resources.ResourceHandleImpl$$Lambda$101/0x000000010017f840.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.ResourceHandleImpl.as(ResourceHandleImpl.java:66)
        at com.zfabrik.impl.resources.NamespaceImpl.lambda$resolve$0(NamespaceImpl.java:229)
        at com.zfabrik.impl.resources.NamespaceImpl$$Lambda$100/0x000000010017bc40.get(Unknown Source)
        at com.zfabrik.work.CycleProtector.supply(CycleProtector.java:117)
        at com.zfabrik.impl.resources.NamespaceImpl.resolve(NamespaceImpl.java:201)
        at com.zfabrik.impl.resources.NamespaceImpl.lookup(NamespaceImpl.java:245)
        at com.zfabrik.components.IComponentsLookup$initializer$1.lookup(IComponentsLookup.java:72)
        at com.tracekey.mytk.bpm.foundation.api.domain.connectors.Connectors$WorkUnitConnector$ConnectorOnDemand.getConnector(Connectors.java:122)
        at com.tracekey.mytk.bpm.foundation.api.domain.connectors.Connectors$WorkUnitConnector$$Lambda$915/0x0000000101002040.apply(Unknown Source)
        at java.util.Optional.map(java.base@11.0.21/Optional.java:265)
        at com.tracekey.mytk.bpm.foundation.api.domain.connectors.Connectors$WorkUnitConnector.getConnector(Connectors.java:162)
        at com.tracekey.mytk.bpm.foundation.api.domain.connectors.Connectors.getConnector(Connectors.java:38)
        at com.tracekey.mytk.bpm.foundation.api.domain.AbstractBusinessPartnerPO.getConnector(AbstractBusinessPartnerPO.java:209)
        at com.tracekey.connectors.russia.foundation.api.resource.explorer.RussianFileExplorer.lambda$8(RussianFileExplorer.java:184)
        at com.tracekey.connectors.russia.foundation.api.resource.explorer.RussianFileExplorer$$Lambda$1183/0x0000000100ff5c40.get(Unknown Source)
        at com.tracekey.mytk.util.impl.TxUtilImpl.supply_aroundBody2(TxUtilImpl.java:26)
        at com.tracekey.mytk.util.impl.TxUtilImpl$AjcClosure3.run(TxUtilImpl.java:1)
        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:67)
        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:73)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:71)
        at com.tracekey.mytk.util.impl.TxUtilImpl.supply(TxUtilImpl.java:26)
        at com.tracekey.mytk.util.api.tx.TransactionUtil.supply(TransactionUtil.java:45)
        at com.tracekey.connectors.russia.foundation.api.resource.explorer.RussianFileExplorer.checkOnCommunicationPartner(RussianFileExplorer.java:184)
        at com.tracekey.connectors.russia.foundation.impl.receiving.RussianReceiveExplorer.access$0(RussianReceiveExplorer.java:1)
        at com.tracekey.connectors.russia.foundation.impl.receiving.RussianReceiveExplorer$1.check(RussianReceiveExplorer.java:58)
        at com.tracekey.mytk.util.imports.impl.service.ImportServiceImpl.lambda$2(ImportServiceImpl.java:91)
        at com.tracekey.mytk.util.imports.impl.service.ImportServiceImpl$$Lambda$791/0x0000000100df6040.run(Unknown Source)
        at com.zfabrik.work.WorkUnit.lambda$work$0(WorkUnit.java:357)
        at com.zfabrik.work.WorkUnit$$Lambda$792/0x0000000100df6440.run(Unknown Source)
        at com.zfabrik.work.WorkUnit.lambda$run$1(WorkUnit.java:365)
        at com.zfabrik.work.WorkUnit$$Lambda$793/0x0000000100df5840.get(Unknown Source)
        at com.zfabrik.work.WorkUnit.supply(WorkUnit.java:378)
        at com.zfabrik.work.WorkUnit.run(WorkUnit.java:365)
        at com.zfabrik.work.WorkUnit.work(WorkUnit.java:357)
        at com.tracekey.mytk.util.imports.impl.service.ImportServiceImpl.lambda$1(ImportServiceImpl.java:89)
        at com.tracekey.mytk.util.imports.impl.service.ImportServiceImpl$$Lambda$790/0x0000000100df6c40.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.21/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.21/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@11.0.21/Thread.java:829)

Found 1 deadlock.
#1

Updated by Henning Blohm 6 days ago

  • Description updated (diff)
#2

Updated by Henning Blohm 6 days ago

Analysis

The problem here is that we may have the following situation:

  • A dependency graph: A->B.
  • Thread T1 wants to access A.as(..) finds that it needs to be build, calls make(...) which locks on the Component Builder (CB). Now T1 has A and CB.
  • Thread T1 wants to access B.as(...). That locks on B. T1 finds that B needs to be build but has to wait for CB.
  • T1 on the other hand wants to access B to resolve the dependency and so needs to wait for B.

Solution

Components builder should not lock and call out to other resources while holding the lock. Instead, CB should first resolve the dependencies and then only lock to protect the compiler and other shared resources.

#3

Updated by Henning Blohm 6 days ago

  • Category set to z2-core
  • Status changed from New to In Progress
  • Assignee set to Henning Blohm
  • Priority changed from Normal to High
  • Target version set to 2.10.2

Also available in: Atom PDF