Jobsub ID 252708.59@dunegpschedd02.fnal.gov
| Jobsub ID | 252708.59@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 05:41:08 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 07:20:44 |
| From worker node | Hostname | ccwcondor0054 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 05:43:31 |
| Input files | hd-protodune:pdhd_prod_beam__244252_167_1_20251106T110310Z_gen_g4_IonScintPDExt.root_249487_138_1_20251121T235554Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5809s) |
| CPU time | 1h (5301s = 91%) |
| Max RSS bytes | 6864572416 (6546 MiB) |
| Outputting started | 2025-11-26 07:20:21 |
| Output files | |
| Finished | 2025-11-26 07:20:44 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
r: q=3.24463e+06 n=1261642 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9623 XUs and 8723 XVs -> 75 XUVs
C:0 T:1 66472 XUs and 63036 XVs -> 3073 XUVs
C:0 T:2 25022 XUs and 26140 XVs -> 392 XUVs
C:0 T:3 7 XUs and 30 XVs -> 0 XUVs
C:0 T:5 43260 XUs and 82716 XVs -> 2415 XUVs
C:0 T:6 48482 XUs and 63735 XVs -> 2270 XUVs
C:0 T:7 7 XUs and 8 XVs -> 0 XUVs
8225 XUVs total
2972 collection wire objects
8225 potential space points
Neighbour search...
479303 tests to find 244768 neighbours
Iterating with no regularization...
Begin: 4.35115e+07
0 3.93109e+07
1 3.88839e+07
2 3.88169e+07
3 3.87958e+07
Now with regularization...
Begin: 3.59998e+07
0 3.59804e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 08:20:15 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root"
26-Nov-2025 08:20:15 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 85.235 103.128 134.294 92.755 17.7785 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00117589 0.00678955 0.0118275 0.00748819 0.00342983 10
reco:opdec:Deconvolution 0.588471 0.726283 0.826852 0.724463 0.0815057 10
reco:ophitspe:OpHitFinderDeco 0.106516 0.133352 0.199609 0.118914 0.0305007 10
reco:opflash:OpFlashFinder 0.0581618 0.0827533 0.101828 0.0855854 0.0144621 10
reco:opslicer:OpSlicer 0.157459 0.222832 0.283271 0.233734 0.0380767 10
reco:wclsdatahd:WireCellToolkit 37.0665 40.4931 46.63 39.9027 2.8949 10
reco:gaushit:GausHitFinder 0.446754 0.684117 0.902061 0.675694 0.1666 10
reco:nhitsfilter:NumberOfHitsFilter 0.00020344 0.000345196 0.00078863 0.000273276 0.000175056 10
reco:reco3d:SpacePointSolver 0.351728 0.783398 1.37722 0.812695 0.290413 10
reco:hitpdune:DisambigFromSpacePoints 0.264347 0.530268 0.767033 0.55029 0.165437 10
reco:pandora:StandardPandora 10.2086 19.4623 37.7129 18.6195 8.00823 10
reco:pandoraWriter:StandardPandora 0.161799 0.241746 0.363327 0.22321 0.0607244 10
reco:pandoraTrack:LArPandoraTrackCreation 0.824543 1.72694 3.62285 1.49049 0.911706 10
reco:pandoraShower:LArPandoraShowerCreation 0.632795 1.49944 2.82591 1.18814 0.778327 10
reco:pandoracalo:Calorimetry 0.516521 0.768776 1.46113 0.636647 0.288758 10
reco:pandoracalonosce:Calorimetry 0.424086 0.797073 1.54793 0.668494 0.353784 10
reco:pandoraShowercalo:ShowerCalorimetry 0.490011 1.44517 2.87501 1.16105 0.79221 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.483199 1.3239 2.75192 1.13932 0.67326 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0233085 0.0397573 0.0688617 0.0361466 0.0129444 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.023251 0.0411695 0.0704164 0.0389208 0.013391 10
reco:emtrkmichelid:EmTrackMichelId 13.656 17.8508 22.087 18.1044 2.4478 10
reco:crttag:SingleCRTMatchingProducer 0.00368357 0.00669765 0.0123553 0.00600559 0.00247559 10
reco:crtreco:TwoCRTMatchingProducer 0.00333115 0.00522094 0.00947021 0.00443239 0.00181893 10
reco:pandora2Track:LArPandoraTrackCreation 2.19839 4.53212 6.96254 3.95686 1.66089 10
reco:pandora2calo:ShowerCalorimetry 0.517903 1.29941 2.62698 1.04009 0.685249 10
reco:pandora2calonosce:ShowerCalorimetry 0.613231 1.28021 2.91948 1.02767 0.698222 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0310951 0.0433689 0.0667651 0.0393065 0.011327 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0303716 0.0396489 0.0624349 0.0363277 0.0093319 10
reco:pandora2Shower:LArPandoraShowerCreation 0.766419 1.65054 3.26338 1.56997 0.672847 10
reco:pandora2Showercalo:ShowerCalorimetry 0.733296 1.66792 3.05171 1.50174 0.653398 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.740097 1.68278 3.06002 1.36319 0.685047 10
[art]:TriggerResults:TriggerResultInserter 2.3095e-05 4.36873e-05 0.00012503 3.593e-05 2.78121e-05 10
end_path:out1:RootOutput 5.999e-06 1.07225e-05 3.4954e-05 7.892e-06 8.19817e-06 10
end_path:out1:RootOutput(write) 1.43923 1.90255 2.39861 1.91959 0.238367 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6696.85 MB
Peak resident set size usage (VmHWM): 4125.52 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.temp.json ====
.:
total 6602060
-rw-r--r-- 1 dune001 lbno 41386 Nov 26 08:20 jobscript.log
-rw-r--r-- 1 dune001 lbno 205 Nov 26 08:20 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:20 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:20 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 08:20 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 08:20 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:20 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 08:20 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 08:20 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 08:20 time.db
-rw-r--r-- 1 dune001 lbno 6707894481 Nov 26 08:20 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252708_59_1_20251126T054336Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 52300070 Nov 26 08:19 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:03 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:01 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:01 detsim_single_protodunehd_hist.root 