Jobsub ID 258430.172@dunegpschedd01.fnal.gov
| Jobsub ID | 258430.172@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 07:19:14 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:53:36 |
| From worker node | Hostname | ccwcondor0054 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 07:22:35 |
| Input files | hd-protodune:pdhd_prod_beam__238172_75_1_20251106T034746Z_gen_g4_IonScintPDExt.root_255160_86_1_20251121T235912Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5424s) |
| CPU time | 1h (4954s = 91%) |
| Max RSS bytes | 6480117760 (6179 MiB) |
| Outputting started | 2025-11-26 08:53:00 |
| Output files | |
| Finished | 2025-11-26 08:53:36 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 13231 XUs and 9654 XVs -> 213 XUVs
C:0 T:1 115181 XUs and 86653 XVs -> 4236 XUVs
C:0 T:2 26100 XUs and 24233 XVs -> 697 XUVs
C:0 T:4 130 XUs and 69 XVs -> 0 XUVs
C:0 T:5 43611 XUs and 39661 XVs -> 1671 XUVs
C:0 T:6 17053 XUs and 11560 XVs -> 425 XUVs
7242 XUVs total
2804 collection wire objects
7242 potential space points
Neighbour search...
233996 tests to find 122900 neighbours
Iterating with no regularization...
Begin: 3.32056e+07
0 3.09115e+07
1 3.0726e+07
2 3.07031e+07
Now with regularization...
Begin: 2.90633e+07
0 2.90567e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 09:52:54 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root"
26-Nov-2025 09:52:54 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 87.8613 123.7 187.398 108.493 36.0667 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00117923 0.00965457 0.0304733 0.00768514 0.00818315 10
reco:opdec:Deconvolution 0.702293 0.816133 0.966876 0.803992 0.0912255 10
reco:ophitspe:OpHitFinderDeco 0.170708 0.201309 0.259428 0.195208 0.0229068 10
reco:opflash:OpFlashFinder 0.0575894 0.0854663 0.137454 0.0772461 0.0249609 10
reco:opslicer:OpSlicer 0.181736 0.251844 0.315594 0.243918 0.0386096 10
reco:wclsdatahd:WireCellToolkit 42.3738 49.9762 55.5002 51.102 3.96006 10
reco:gaushit:GausHitFinder 0.5471 0.852793 1.28754 0.843602 0.20641 10
reco:nhitsfilter:NumberOfHitsFilter 0.000210971 0.000337311 0.000675527 0.000320202 0.000133453 10
reco:reco3d:SpacePointSolver 0.364439 1.0769 2.71791 0.827994 0.747368 10
reco:hitpdune:DisambigFromSpacePoints 0.244832 0.630164 1.22383 0.544104 0.308103 10
reco:pandora:StandardPandora 9.70225 24.0274 66.8937 14.2345 17.5315 10
reco:pandoraWriter:StandardPandora 0.193079 0.30177 0.490243 0.297083 0.0838955 10
reco:pandoraTrack:LArPandoraTrackCreation 0.953738 2.03347 4.80938 1.72639 1.11155 10
reco:pandoraShower:LArPandoraShowerCreation 0.785839 2.07208 5.6368 1.7854 1.39803 10
reco:pandoracalo:Calorimetry 0.515879 0.969338 2.214 0.797386 0.517336 10
reco:pandoracalonosce:Calorimetry 0.485763 0.981598 2.31577 0.796108 0.5397 10
reco:pandoraShowercalo:ShowerCalorimetry 0.759906 1.94876 5.27554 1.55928 1.28226 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.689748 1.82437 4.34332 1.53499 1.08532 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0289416 0.0585199 0.0933729 0.0557057 0.0215576 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0295156 0.0578417 0.0898646 0.0542696 0.0206432 10
reco:emtrkmichelid:EmTrackMichelId 14.594 19.0527 27.7075 18.4793 3.92869 10
reco:crttag:SingleCRTMatchingProducer 0.00481833 0.00695807 0.0120209 0.00626912 0.00195897 10
reco:crtreco:TwoCRTMatchingProducer 0.00341844 0.00562047 0.0114242 0.00490111 0.00227301 10
reco:pandora2Track:LArPandoraTrackCreation 2.2468 4.98212 12.3195 4.24618 2.94741 10
reco:pandora2calo:ShowerCalorimetry 0.727613 1.56412 3.80036 1.40565 0.879218 10
reco:pandora2calonosce:ShowerCalorimetry 0.521444 1.48794 3.9843 1.25032 0.960025 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0281938 0.0578348 0.0910159 0.0552769 0.017034 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0278862 0.0565823 0.0913848 0.0546361 0.017496 10
reco:pandora2Shower:LArPandoraShowerCreation 0.604089 1.94223 4.72406 1.70354 1.12693 10
reco:pandora2Showercalo:ShowerCalorimetry 0.661011 2.06457 5.43516 1.66028 1.30728 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.928245 1.95071 5.03314 1.6309 1.18457 10
[art]:TriggerResults:TriggerResultInserter 3.2409e-05 4.80482e-05 0.000119482 4.11625e-05 2.41635e-05 10
end_path:out1:RootOutput 7.912e-06 1.17698e-05 3.5143e-05 8.8585e-06 7.86387e-06 10
end_path:out1:RootOutput(write) 1.64415 2.18365 3.07004 2.18463 0.428249 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6632.71 MB
Peak resident set size usage (VmHWM): 4033.66 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.temp.json ====
.:
total 5772772
-rw-r--r-- 1 dune001 lbno 40509 Nov 26 09:53 jobscript.log
-rw-r--r-- 1 dune001 lbno 203 Nov 26 09:53 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:52 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:52 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:52 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:52 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:52 time.db
-rw-r--r-- 1 dune001 lbno 5862568162 Nov 26 09:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258430_172_1_20251126T072239Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 48436100 Nov 26 09:51 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:32 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:30 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:30 detsim_single_protodunehd_hist.root 