Jobsub ID 252429.130@dunegpschedd02.fnal.gov
| Jobsub ID | 252429.130@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 01:45:56 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 04:23:52 |
| From worker node | Hostname | n161.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 01:50:21 |
| Input files | hd-protodune:pdhd_prod_beam__238220_139_1_20251106T050804Z_gen_g4_IonScintPDExt.root_255234_105_1_20251122T074338Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (9119s) |
| CPU time | 1h (4988s = 54%) |
| Max RSS bytes | 7018061824 (6692 MiB) |
| Outputting started | 2025-11-26 04:22:22 |
| Output files | |
| Finished | 2025-11-26 04:23:52 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
6 XVs -> 0 XUVs
3816 XUVs total
2155 collection wire objects
3816 potential space points
Neighbour search...
80216 tests to find 36228 neighbours
Iterating with no regularization...
Begin: 5.1582e+07
0 4.94974e+07
1 4.93773e+07
2 4.93665e+07
Now with regularization...
Begin: 4.70173e+07
0 4.70146e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00289124 0.000207126 0.000207186 0.000463724 0.000126481 0.000126421 0.000126481 0.000126421 0.000126481 0.000482917 0.000158966 0.000158906 0.00134373
----view 1: 6.4671e-05 6.46114e-05 6.4671e-05 6.4671e-05 6.46114e-05 8.17776e-05 0.000277281 0.000277221 0.00025785 0.000222623 0.000222623 0.000222623 0.000222623
26-Nov-2025 04:22:16 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root"
26-Nov-2025 04:22:16 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 121.731 162.405 206.488 166.25 25.4189 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000857943 0.00842072 0.0217364 0.0073146 0.00570219 10
reco:opdec:Deconvolution 1.01823 1.16669 1.267 1.17006 0.0696511 10
reco:ophitspe:OpHitFinderDeco 0.227387 0.244718 0.30941 0.238911 0.0224456 10
reco:opflash:OpFlashFinder 0.0886757 0.115369 0.157403 0.113683 0.0203977 10
reco:opslicer:OpSlicer 0.241504 0.294415 0.339599 0.296371 0.028591 10
reco:wclsdatahd:WireCellToolkit 58.7717 60.9736 63.8197 61.0783 1.58702 10
reco:gaushit:GausHitFinder 0.695818 0.992901 1.18875 0.968746 0.144986 10
reco:nhitsfilter:NumberOfHitsFilter 0.000363963 0.000500164 0.000685091 0.000494007 8.15719e-05 10
reco:reco3d:SpacePointSolver 0.39641 1.15965 2.41072 1.02988 0.521529 10
reco:hitpdune:DisambigFromSpacePoints 0.511145 0.928959 1.38771 0.919129 0.258642 10
reco:pandora:StandardPandora 14.96 31.5002 54.3583 28.2392 12.0215 10
reco:pandoraWriter:StandardPandora 0.233419 0.318279 0.38659 0.324679 0.0425562 10
reco:pandoraTrack:LArPandoraTrackCreation 1.98883 3.81898 5.28505 4.09573 1.14681 10
reco:pandoraShower:LArPandoraShowerCreation 1.2745 3.45938 5.20449 3.29759 1.30092 10
reco:pandoracalo:Calorimetry 0.877208 1.47402 1.96144 1.52434 0.378942 10
reco:pandoracalonosce:Calorimetry 0.89173 1.47521 1.93139 1.5239 0.35716 10
reco:pandoraShowercalo:ShowerCalorimetry 1.08742 2.65681 3.973 2.58649 0.935194 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.0901 2.50685 3.62616 2.43063 0.796939 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0460918 0.0594991 0.0693368 0.0592954 0.0057806 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.046512 0.0595452 0.0692834 0.0593934 0.00580268 10
reco:emtrkmichelid:EmTrackMichelId 17.6363 24.4589 28.7927 25.134 3.15319 10
reco:crttag:SingleCRTMatchingProducer 0.00978518 0.0114527 0.0128574 0.0116706 0.00105619 10
reco:crtreco:TwoCRTMatchingProducer 0.00742269 0.00981657 0.0115939 0.0101381 0.00141625 10
reco:pandora2Track:LArPandoraTrackCreation 4.54949 8.65087 12.0253 8.57687 2.63586 10
reco:pandora2calo:ShowerCalorimetry 0.965869 2.25395 3.33108 2.22807 0.698837 10
reco:pandora2calonosce:ShowerCalorimetry 1.07227 2.15363 3.19953 2.08305 0.664516 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0449862 0.0566209 0.0659894 0.0563042 0.00610153 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0450696 0.0570717 0.0656987 0.0568482 0.00588839 10
reco:pandora2Shower:LArPandoraShowerCreation 1.54356 3.19052 4.58117 3.18392 1.03118 10
reco:pandora2Showercalo:ShowerCalorimetry 1.53388 2.96617 4.15354 2.85972 0.86538 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.44848 2.83786 3.88594 2.87203 0.783878 10
[art]:TriggerResults:TriggerResultInserter 2.5049e-05 3.49657e-05 7.4211e-05 3.0633e-05 1.34216e-05 10
end_path:out1:RootOutput 6.08e-06 8.0961e-06 1.9801e-05 6.734e-06 3.93132e-06 10
end_path:out1:RootOutput(write) 1.91243 2.40797 2.66856 2.45365 0.226103 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6870.33 MB
Peak resident set size usage (VmHWM): 4361.24 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.temp.json ====
.:
total 6106696
-rw-r--r--. 1 dune004 dune 41165 Nov 26 04:22 jobscript.log
-rw-r--r--. 1 dune004 dune 205 Nov 26 04:22 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 04:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 04:22 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 04:22 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 04:22 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 04:22 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 04:22 time.db
-rw-r--r--. 1 dune004 dune 6202347190 Nov 26 04:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252429_130_1_20251126T015025Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 50579450 Nov 26 04:21 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:56 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:53 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 03:52 detsim_single_protodunehd_hist.root 