Jobsub ID 252723.17@dunegpschedd02.fnal.gov
| Jobsub ID | 252723.17@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:03:08 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 07:53:02 |
| From worker node | Hostname | ccwcondor0003 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:04:12 |
| Input files | hd-protodune:pdhd_prod_beam__244153_24_1_20251106T090256Z_gen_g4_IonScintPDExt.root_250348_4_1_20251124T034229Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6503s) |
| CPU time | 1h (6330s = 97%) |
| Max RSS bytes | 8844177408 (8434 MiB) |
| Outputting started | 2025-11-26 07:52:36 |
| Output files | |
| Finished | 2025-11-26 07:53:02 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
Us and 134 XVs -> 0 XUVs
C:0 T:5 20300 XUs and 16035 XVs -> 374 XUVs
C:0 T:6 35709 XUs and 39430 XVs -> 999 XUVs
C:0 T:7 3 XUs and 18 XVs -> 0 XUVs
8158 XUVs total
4005 collection wire objects
8158 potential space points
Neighbour search...
306608 tests to find 113006 neighbours
Iterating with no regularization...
Begin: 6.00328e+07
0 5.29367e+07
1 5.24584e+07
2 5.24054e+07
3 5.23899e+07
Now with regularization...
Begin: 5.07781e+07
0 5.0769e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 08:52:30 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root"
26-Nov-2025 08:52:30 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 108.453 159.777 227.213 150.065 39.9476 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00136039 0.0101832 0.0208685 0.00980383 0.00628431 10
reco:opdec:Deconvolution 0.616354 0.896974 1.23465 0.928526 0.187954 10
reco:ophitspe:OpHitFinderDeco 0.115548 0.167308 0.224685 0.167285 0.0473928 10
reco:opflash:OpFlashFinder 0.0710714 0.109806 0.190628 0.0967771 0.0350015 10
reco:opslicer:OpSlicer 0.176171 0.272544 0.409862 0.257774 0.0773228 10
reco:wclsdatahd:WireCellToolkit 40.155 44.6677 48.7148 44.0035 3.14918 10
reco:gaushit:GausHitFinder 0.542932 0.913881 1.47505 0.884892 0.266382 10
reco:nhitsfilter:NumberOfHitsFilter 0.00037615 0.000725151 0.00204176 0.000536834 0.000493318 10
reco:reco3d:SpacePointSolver 0.60019 1.36971 2.48741 1.34273 0.587161 10
reco:hitpdune:DisambigFromSpacePoints 0.50358 0.800661 1.36902 0.78551 0.253517 10
reco:pandora:StandardPandora 16.0131 31.7197 58.0229 26.8346 12.8774 10
reco:pandoraWriter:StandardPandora 0.263847 0.367069 0.509571 0.341686 0.0876373 10
reco:pandoraTrack:LArPandoraTrackCreation 1.3841 4.47625 9.5899 3.70349 2.48768 10
reco:pandoraShower:LArPandoraShowerCreation 1.5296 4.64857 7.85091 4.34345 2.16117 10
reco:pandoracalo:Calorimetry 0.81875 1.97315 3.94043 1.54542 1.07478 10
reco:pandoracalonosce:Calorimetry 0.923611 1.74111 2.98398 1.37201 0.8031 10
reco:pandoraShowercalo:ShowerCalorimetry 1.91116 3.82123 5.68367 3.69981 1.30868 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.36691 3.45084 6.82778 3.26115 1.64641 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.043493 0.0646685 0.0920404 0.0615484 0.0171392 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0409955 0.0612571 0.0912869 0.0571476 0.0175747 10
reco:emtrkmichelid:EmTrackMichelId 19.8377 23.8256 28.7551 22.9004 2.93852 10
reco:crttag:SingleCRTMatchingProducer 0.00728763 0.0114917 0.0173785 0.0110757 0.00299067 10
reco:crtreco:TwoCRTMatchingProducer 0.0053091 0.00991597 0.0171365 0.00912511 0.0032703 10
reco:pandora2Track:LArPandoraTrackCreation 5.90526 12.3562 19.4249 12.399 5.04621 10
reco:pandora2calo:ShowerCalorimetry 1.26927 3.6353 6.76624 3.74495 1.72267 10
reco:pandora2calonosce:ShowerCalorimetry 1.25781 2.95614 4.08082 3.52998 1.06164 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.04266 0.067342 0.0951903 0.0659294 0.0165305 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0411265 0.0621816 0.088524 0.0640902 0.0140348 10
reco:pandora2Shower:LArPandoraShowerCreation 2.36317 4.4088 7.26323 4.35785 1.87925 10
reco:pandora2Showercalo:ShowerCalorimetry 2.11573 4.49457 7.99701 4.40359 1.98877 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.60217 3.86534 6.65975 3.77267 1.85962 10
[art]:TriggerResults:TriggerResultInserter 2.9153e-05 5.43348e-05 0.000113038 4.4246e-05 2.53529e-05 10
end_path:out1:RootOutput 6.509e-06 1.47609e-05 6.9623e-05 8.723e-06 1.83484e-05 10
end_path:out1:RootOutput(write) 1.78024 2.3156 3.19491 2.16915 0.460213 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7767.4 MB
Peak resident set size usage (VmHWM): 5185.9 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.temp.json ====
.:
total 7006208
-rw-r--r-- 1 dune001 lbno 61508 Nov 26 08:52 jobscript.log
-rw-r--r-- 1 dune001 lbno 203 Nov 26 08:52 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 08:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 08:52 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:52 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 08:52 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 08:52 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 08:52 time.db
-rw-r--r-- 1 dune001 lbno 7111857535 Nov 26 08:52 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252723_17_1_20251126T060417Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 62162890 Nov 26 08:50 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:26 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:24 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:23 detsim_single_protodunehd_hist.root 