Jobsub ID 258403.17@dunegpschedd01.fnal.gov
| Jobsub ID | 258403.17@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:29:09 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:22:16 |
| From worker node | Hostname | ccwcondor0551 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:30:15 |
| Input files | hd-protodune:pdhd_prod_beam__238225_162_1_20251106T051752Z_gen_g4_IonScintPDExt.root_255163_2_1_20251122T001315Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6690s) |
| CPU time | 1h (6073s = 90%) |
| Max RSS bytes | 6740766720 (6428 MiB) |
| Outputting started | 2025-11-26 08:21:46 |
| Output files | |
| Finished | 2025-11-26 08:22:16 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12197 XUs and 9140 XVs -> 212 XUVs
C:0 T:1 56260 XUs and 53261 XVs -> 3077 XUVs
C:0 T:2 12971 XUs and 8909 XVs -> 415 XUVs
C:0 T:5 44068 XUs and 37179 XVs -> 1228 XUVs
C:0 T:6 16614 XUs and 54686 XVs -> 1500 XUVs
C:0 T:7 1 XUs and 4 XVs -> 0 XUVs
6432 XUVs total
2422 collection wire objects
6432 potential space points
Neighbour search...
535158 tests to find 268262 neighbours
Iterating with no regularization...
Begin: 3.3062e+07
0 3.0517e+07
1 3.0398e+07
2 3.03852e+07
Now with regularization...
Begin: 2.89971e+07
0 2.89913e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 09:21:39 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root"
26-Nov-2025 09:21:39 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 123.334 168.702 204.524 171.615 27.8854 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00192555 0.0077604 0.0117491 0.00888367 0.00315963 10
reco:opdec:Deconvolution 0.638023 0.911928 1.10686 0.937054 0.147856 10
reco:ophitspe:OpHitFinderDeco 0.135295 0.233398 0.27173 0.240721 0.0349131 10
reco:opflash:OpFlashFinder 0.0644418 0.0911435 0.128595 0.0906176 0.0215849 10
reco:opslicer:OpSlicer 0.194438 0.287129 0.368715 0.298588 0.0568999 10
reco:wclsdatahd:WireCellToolkit 48.6613 50.4075 53.6943 50.0324 1.4849 10
reco:gaushit:GausHitFinder 0.634192 1.10385 1.4297 1.14464 0.203897 10
reco:nhitsfilter:NumberOfHitsFilter 0.000320289 0.000465636 0.00074349 0.000443083 0.000106293 10
reco:reco3d:SpacePointSolver 0.315751 1.38242 2.68753 1.21019 0.746016 10
reco:hitpdune:DisambigFromSpacePoints 0.260891 0.778007 1.2709 0.758604 0.28669 10
reco:pandora:StandardPandora 14.0142 32.512 46.3887 30.4478 10.2084 10
reco:pandoraWriter:StandardPandora 0.284698 0.333826 0.422019 0.315629 0.0475146 10
reco:pandoraTrack:LArPandoraTrackCreation 2.25765 3.69761 4.92658 3.70943 0.876118 10
reco:pandoraShower:LArPandoraShowerCreation 1.93954 3.58257 5.32012 3.64847 1.1119 10
reco:pandoracalo:Calorimetry 1.31396 1.75697 2.41348 1.71512 0.37205 10
reco:pandoracalonosce:Calorimetry 1.16468 1.68703 2.20236 1.60964 0.321715 10
reco:pandoraShowercalo:ShowerCalorimetry 1.67593 3.29383 4.41203 3.32822 0.931396 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.79812 3.2384 5.6399 3.08998 1.12194 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0373899 0.073909 0.0997604 0.0749199 0.0159209 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0360432 0.0714463 0.0986757 0.0754936 0.0165899 10
reco:emtrkmichelid:EmTrackMichelId 28.9421 36.7967 44.4642 37.5389 5.45703 10
reco:crttag:SingleCRTMatchingProducer 0.00857393 0.0114679 0.0150382 0.0109891 0.00191527 10
reco:crtreco:TwoCRTMatchingProducer 0.00737377 0.0101397 0.0147299 0.00944229 0.00230017 10
reco:pandora2Track:LArPandoraTrackCreation 4.69773 8.35368 12.0313 8.40419 2.62764 10
reco:pandora2calo:ShowerCalorimetry 0.940874 2.72287 4.99706 2.64568 1.24748 10
reco:pandora2calonosce:ShowerCalorimetry 0.938811 2.4536 3.68757 2.37927 0.893183 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0519304 0.0701018 0.0919488 0.0703865 0.0115287 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0478517 0.0686631 0.0890634 0.0675245 0.0118591 10
reco:pandora2Shower:LArPandoraShowerCreation 2.00176 3.51522 4.90095 3.64087 0.992034 10
reco:pandora2Showercalo:ShowerCalorimetry 1.83941 3.33206 5.21974 3.61552 1.11571 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.61058 3.24261 4.67643 3.43044 1.03432 10
[art]:TriggerResults:TriggerResultInserter 4.7489e-05 7.13744e-05 0.00013357 6.5462e-05 2.43545e-05 10
end_path:out1:RootOutput 7.514e-06 1.29851e-05 3.765e-05 9.853e-06 8.57067e-06 10
end_path:out1:RootOutput(write) 1.68962 2.46691 2.88327 2.57712 0.408354 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6312.17 MB
Peak resident set size usage (VmHWM): 4107.25 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.temp.json ====
.:
total 5994384
-rw-r--r-- 1 dune001 lbno 41026 Nov 26 09:21 jobscript.log
-rw-r--r-- 1 dune001 lbno 203 Nov 26 09:21 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:21 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:21 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:21 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:21 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:21 time.db
-rw-r--r-- 1 dune001 lbno 6085049653 Nov 26 09:21 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258403_17_1_20251126T063020Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 52883620 Nov 26 09:20 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:54 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:51 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:51 detsim_single_protodunehd_hist.root 