Jobsub ID 258334.18@dunegpschedd01.fnal.gov
| Jobsub ID | 258334.18@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 04:55:05 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 08:52:12 |
| From worker node | Hostname | wn121.cat.cbpf.br |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 04:56:36 |
| Input files | hd-protodune:pdhd_prod_beam__238386_50_1_20251106T072853Z_gen_g4_IonScintPDExt.root_255412_191_1_20251122T175422Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (13879s) |
| CPU time | 2h (9274s = 66%) |
| Max RSS bytes | 4259205120 (4061 MiB) |
| Outputting started | 2025-11-26 08:47:57 |
| Output files | |
| Finished | 2025-11-26 08:52:12 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
races tagged "wiener"
FrameSaver: q=3.63187e+06 n=1496036 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 20571 XUs and 15154 XVs -> 202 XUVs
C:0 T:1 111057 XUs and 75928 XVs -> 2739 XUVs
C:0 T:2 35342 XUs and 35113 XVs -> 991 XUVs
C:0 T:4 31 XUs and 12 XVs -> 0 XUVs
C:0 T:5 49925 XUs and 52947 XVs -> 652 XUVs
C:0 T:6 20012 XUs and 21012 XVs -> 584 XUVs
5168 XUVs total
3254 collection wire objects
5168 potential space points
Neighbour search...
116242 tests to find 51412 neighbours
Iterating with no regularization...
Begin: 4.81674e+07
0 4.58512e+07
1 4.57138e+07
2 4.57057e+07
Now with regularization...
Begin: 4.38186e+07
0 4.38133e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 05:47:47 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root"
26-Nov-2025 05:47:47 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 178.726 257.787 427.407 239.699 70.5554 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00150656 0.0110376 0.0199374 0.0108064 0.00581724 10
reco:opdec:Deconvolution 1.07041 1.42651 1.8227 1.47381 0.219505 10
reco:ophitspe:OpHitFinderDeco 0.179593 0.304502 0.3454 0.314538 0.0443156 10
reco:opflash:OpFlashFinder 0.0911914 0.16093 0.248943 0.154301 0.0465535 10
reco:opslicer:OpSlicer 0.329147 0.45562 0.65779 0.476057 0.0928933 10
reco:wclsdatahd:WireCellToolkit 66.0468 74.3816 79.5934 73.5559 4.1227 10
reco:gaushit:GausHitFinder 1.03709 1.62881 2.34095 1.63952 0.352804 10
reco:nhitsfilter:NumberOfHitsFilter 0.000422477 0.000659352 0.000907846 0.000672601 0.000142112 10
reco:reco3d:SpacePointSolver 0.660126 1.73063 4.12185 1.30791 1.0776 10
reco:hitpdune:DisambigFromSpacePoints 0.426218 1.58616 3.52113 1.49186 0.814892 10
reco:pandora:StandardPandora 23.1305 52.8531 112.58 44.2043 27.9173 10
reco:pandoraWriter:StandardPandora 0.329641 0.475319 0.779895 0.425926 0.147619 10
reco:pandoraTrack:LArPandoraTrackCreation 1.96083 3.83608 6.68186 3.58264 1.37919 10
reco:pandoraShower:LArPandoraShowerCreation 2.06339 4.34102 10.4371 4.28639 2.36081 10
reco:pandoracalo:Calorimetry 1.11568 2.24771 4.14122 1.91219 0.923433 10
reco:pandoracalonosce:Calorimetry 0.864196 2.13469 3.69698 1.87208 0.85775 10
reco:pandoraShowercalo:ShowerCalorimetry 1.81136 4.54965 11.0841 3.90375 2.63508 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.93236 4.28542 10.4079 3.7537 2.40361 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0866115 0.116012 0.170537 0.115702 0.0240866 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0867149 0.117222 0.171655 0.116203 0.0224161 10
reco:emtrkmichelid:EmTrackMichelId 47.0682 62.0052 102.672 55.9411 16.7092 10
reco:crttag:SingleCRTMatchingProducer 0.00743724 0.0120923 0.0155802 0.0127055 0.00269006 10
reco:crtreco:TwoCRTMatchingProducer 0.00649273 0.00964242 0.0132689 0.00924068 0.00202639 10
reco:pandora2Track:LArPandoraTrackCreation 5.21155 10.9908 21.8928 10.4066 5.49495 10
reco:pandora2calo:ShowerCalorimetry 1.58095 4.01316 9.10918 3.58401 2.28511 10
reco:pandora2calonosce:ShowerCalorimetry 1.604 3.93827 8.86341 3.47409 2.27269 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0510541 0.107891 0.16347 0.110882 0.0296781 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.05193 0.108637 0.162828 0.116042 0.0291609 10
reco:pandora2Shower:LArPandoraShowerCreation 1.66507 4.6408 9.36627 4.45402 2.46008 10
reco:pandora2Showercalo:ShowerCalorimetry 1.69228 5.3478 11.0444 3.79768 3.12831 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.8617 5.24969 10.9491 3.33978 3.23494 10
[art]:TriggerResults:TriggerResultInserter 2.1661e-05 6.74287e-05 0.000395456 3.31265e-05 0.000109436 10
end_path:out1:RootOutput 4.85e-06 2.57641e-05 0.000196908 5.8105e-06 5.71412e-05 10
end_path:out1:RootOutput(write) 3.44907 4.5383 6.42486 4.18422 0.958836 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6289.42 MB
Peak resident set size usage (VmHWM): 3920.27 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.temp.json ====
.:
total 6679708
-rw-r--r-- 1 nobody nobody 40704 Nov 26 05:47 jobscript.log
-rw-r--r-- 1 nobody nobody 204 Nov 26 05:47 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:47 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:47 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 05:47 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 05:47 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 05:47 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 05:47 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 05:47 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 05:47 time.db
-rw-r--r-- 1 nobody nobody 6780670611 Nov 26 05:47 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258334_18_1_20251126T045643Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 59039330 Nov 26 05:44 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:06 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:01 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 05:01 detsim_single_protodunehd_hist.root 