Jobsub ID 252804.23@dunegpschedd02.fnal.gov
| Jobsub ID | 252804.23@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 08:27:15 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce02 |
| Last heartbeat | 2025-11-26 10:12:24 |
| From worker node | Hostname | ccwcondor0034 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 08:28:43 |
| Input files | hd-protodune:pdhd_prod_beam__238322_9_1_20251106T071107Z_gen_g4_IonScintPDExt.root_255782_165_1_20251123T175617Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6196s) |
| CPU time | 1h (5704s = 92%) |
| Max RSS bytes | 6634475520 (6327 MiB) |
| Outputting started | 2025-11-26 10:12:00 |
| Output files | |
| Finished | 2025-11-26 10:12:24 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
wclsFrameSaver: saving 84908 traces tagged "wiener"
FrameSaver: q=2.51379e+06 n=1152042 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 3389 XUs and 1780 XVs -> 6 XUVs
C:0 T:1 24818 XUs and 9784 XVs -> 210 XUVs
C:0 T:2 8288 XUs and 26461 XVs -> 487 XUVs
C:0 T:3 4 XUs and 5 XVs -> 0 XUVs
C:0 T:4 25 XUs and 30 XVs -> 1 XUVs
C:0 T:5 47546 XUs and 32307 XVs -> 1587 XUVs
C:0 T:6 29893 XUs and 41134 XVs -> 1313 XUVs
C:0 T:7 13 XUs and 22 XVs -> 0 XUVs
3604 XUVs total
2524 collection wire objects
3604 potential space points
Neighbour search...
61080 tests to find 32760 neighbours
Iterating with no regularization...
Begin: 4.03992e+07
0 3.83364e+07
1 3.82377e+07
2 3.82277e+07
Now with regularization...
Begin: 3.63446e+07
0 3.63416e+07
26-Nov-2025 11:11:53 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root"
26-Nov-2025 11:11:53 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 99.1822 139.531 178.183 139.022 26.3439 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00126359 0.00878942 0.0146087 0.008701 0.0043655 10
reco:opdec:Deconvolution 0.752589 0.872634 0.957789 0.869973 0.0601941 10
reco:ophitspe:OpHitFinderDeco 0.117904 0.179929 0.21881 0.200994 0.0365008 10
reco:opflash:OpFlashFinder 0.0723758 0.0865613 0.101761 0.0879008 0.00882261 10
reco:opslicer:OpSlicer 0.188755 0.252921 0.322298 0.251343 0.0411392 10
reco:wclsdatahd:WireCellToolkit 46.9349 48.5467 50.997 48.5788 1.14803 10
reco:gaushit:GausHitFinder 0.646058 0.958878 1.10739 1.01246 0.147124 10
reco:nhitsfilter:NumberOfHitsFilter 0.000267947 0.000354357 0.00052036 0.000329305 7.8761e-05 10
reco:reco3d:SpacePointSolver 0.509726 1.39862 3.25208 1.06689 0.865724 10
reco:hitpdune:DisambigFromSpacePoints 0.428077 0.943387 1.58221 0.994953 0.347377 10
reco:pandora:StandardPandora 10.903 29.0412 58.0267 25.4929 14.7377 10
reco:pandoraWriter:StandardPandora 0.275673 0.346624 0.403991 0.35038 0.0456368 10
reco:pandoraTrack:LArPandoraTrackCreation 1.29965 2.67945 4.18763 2.70488 0.897024 10
reco:pandoraShower:LArPandoraShowerCreation 1.2356 2.74615 4.38712 2.94218 0.979349 10
reco:pandoracalo:Calorimetry 0.671096 1.4912 2.38423 1.54094 0.521226 10
reco:pandoracalonosce:Calorimetry 0.635235 1.42833 1.99473 1.57785 0.473781 10
reco:pandoraShowercalo:ShowerCalorimetry 1.00893 2.42172 4.19298 2.3527 0.87983 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.88254 2.55772 5.20528 2.69105 1.1273 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0435362 0.0673076 0.0877925 0.0668659 0.0117192 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0420824 0.0636376 0.0854642 0.0631758 0.0125312 10
reco:emtrkmichelid:EmTrackMichelId 17.2813 22.0013 24.2961 22.4351 2.16872 10
reco:crttag:SingleCRTMatchingProducer 0.00681665 0.009545 0.0150879 0.00887249 0.00239016 10
reco:crtreco:TwoCRTMatchingProducer 0.00501544 0.00771442 0.0138174 0.00678536 0.00253677 10
reco:pandora2Track:LArPandoraTrackCreation 3.02224 6.77975 9.77465 7.66338 2.23642 10
reco:pandora2calo:ShowerCalorimetry 0.80328 2.01216 3.23356 2.09168 0.739966 10
reco:pandora2calonosce:ShowerCalorimetry 0.863456 1.96072 3.31346 2.10929 0.729181 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.05908 0.0706511 0.0835758 0.0684514 0.00835501 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0433609 0.065099 0.0806271 0.0644577 0.0100424 10
reco:pandora2Shower:LArPandoraShowerCreation 1.40949 2.68913 3.78004 2.72452 0.711033 10
reco:pandora2Showercalo:ShowerCalorimetry 1.32042 2.68929 3.71381 2.75223 0.82581 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.06606 2.54946 3.3771 2.93818 0.795672 10
[art]:TriggerResults:TriggerResultInserter 2.9715e-05 4.71455e-05 0.000117088 4.0817e-05 2.40759e-05 10
end_path:out1:RootOutput 7.382e-06 1.31991e-05 3.1377e-05 9.7695e-06 8.37032e-06 10
end_path:out1:RootOutput(write) 1.97954 2.39477 2.82865 2.39602 0.231341 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6708.96 MB
Peak resident set size usage (VmHWM): 4165.75 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.temp.json ====
.:
total 5879948
-rw-r--r-- 1 dune001 lbno 41953 Nov 26 11:11 jobscript.log
-rw-r--r-- 1 dune001 lbno 203 Nov 26 11:11 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 11:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 11:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 11:11 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 11:11 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 11:11 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 11:11 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 11:11 time.db
-rw-r--r-- 1 dune001 lbno 5963555080 Nov 26 11:11 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252804_23_1_20251126T082848Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 57192980 Nov 26 11:11 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:49 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:47 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:46 detsim_single_protodunehd_hist.root 