justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252891.8@dunegpschedd02.fnal.gov

Jobsub ID252891.8@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 10:11:20
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-11-26 12:20:00
From worker nodeHostnameccwcondor0635
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 10:12:36
Input fileshd-protodune:pdhd_prod_beam__238205_101_1_20251106T043434Z_gen_g4_IonScintPDExt.root_255518_132_1_20251122T215059Z_PDInt.root
JobscriptExit code0
Real time2h (7614s)
CPU time1h (6917s = 90%)
Max RSS bytes6890770432 (6571 MiB)
Outputting started2025-11-26 12:19:31
Output files
Finished2025-11-26 12:20:00
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

C:0 T:0 20574 XUs and 17361 XVs -> 255 XUVs
C:0 T:1 170373 XUs and 142274 XVs -> 5331 XUVs
C:0 T:2 74000 XUs and 79462 XVs -> 4621 XUVs
C:0 T:3 11 XUs and 18 XVs -> 0 XUVs
C:0 T:4 31 XUs and 37 XVs -> 0 XUVs
C:0 T:5 47254 XUs and 43361 XVs -> 1198 XUVs
C:0 T:6 22083 XUs and 26830 XVs -> 751 XUVs
C:0 T:7 9 XUs and 8 XVs -> 0 XUVs
12156 XUVs total
4213 collection wire objects
12156 potential space points
Neighbour search...
734300 tests to find 386132 neighbours
Iterating with no regularization...
Begin: 7.8665e+07
0 7.2108e+07
1 7.15976e+07
2 7.15451e+07
Now with regularization...
Begin: 6.9321e+07
0 6.93e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 13:19:24 CET  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root"
26-Nov-2025 13:19:24 CET  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            130.433       210.239       281.794       204.684       54.3349        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00132921    0.00822704     0.0153276    0.00838725    0.00484423       10     
reco:opdec:Deconvolution                             0.676032       1.07124       1.4738        1.08671      0.200161        10     
reco:ophitspe:OpHitFinderDeco                        0.187477      0.239352      0.279367      0.251905      0.0321311       10     
reco:opflash:OpFlashFinder                           0.0676781      0.11942      0.195843      0.113721      0.032837        10     
reco:opslicer:OpSlicer                               0.205595       0.34714      0.489903      0.362579      0.078025        10     
reco:wclsdatahd:WireCellToolkit                       47.6142       51.5909       54.092        51.9222       2.1088         10     
reco:gaushit:GausHitFinder                           0.605707       1.10224       1.44925       1.14774      0.292093        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000325638   0.000495733   0.000608116   0.000507768   7.49564e-05      10     
reco:reco3d:SpacePointSolver                         0.636305       1.67623       3.40847       1.58367      0.863663        10     
reco:hitpdune:DisambigFromSpacePoints                0.561738       1.2389        2.40614       1.05734      0.602485        10     
reco:pandora:StandardPandora                          18.6748       40.3149       73.4741       34.7712       18.0224        10     
reco:pandoraWriter:StandardPandora                   0.298326      0.420547      0.629163      0.421186      0.0977263       10     
reco:pandoraTrack:LArPandoraTrackCreation             2.86524       5.03115       7.53528       4.19491       1.89882        10     
reco:pandoraShower:LArPandoraShowerCreation           1.70683       5.83717       9.54728       6.55941       2.42052        10     
reco:pandoracalo:Calorimetry                          1.11288       2.55113       4.2774        2.59139       1.10962        10     
reco:pandoracalonosce:Calorimetry                     1.24366       2.58759        4.244        2.43709       1.10056        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.50749       5.64767       12.3187       5.65645       3.05228        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.29299       5.11031       11.3486       4.94965       2.86199        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0503772     0.0953895     0.133421      0.0996236     0.0253859       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0453241     0.0848247      0.12196      0.0869742     0.0225549       10     
reco:emtrkmichelid:EmTrackMichelId                    31.8276       43.4653       59.5754       41.2285       9.37798        10     
reco:crttag:SingleCRTMatchingProducer                0.0084514     0.015454      0.0299782     0.0160956    0.00570154       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00625861     0.0128004     0.0212138     0.0137388    0.00427182       10     
reco:pandora2Track:LArPandoraTrackCreation            4.8999        12.5989       19.1162       11.918        4.96837        10     
reco:pandora2calo:ShowerCalorimetry                   1.33424       4.57695       7.34021       4.70989       2.09673        10     
reco:pandora2calonosce:ShowerCalorimetry              1.51276       4.29179       6.67784       4.31768       1.94002        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0529952     0.0896579      0.12184      0.0940729     0.0216404       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0544404     0.0873218     0.111518      0.0928855     0.0190218       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.58505       5.63468       8.4261        5.59929       2.08335        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.16846       6.01553       9.08337       5.70028       2.56413        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.94673       5.19858       8.77522       5.05499       2.30612        10     
[art]:TriggerResults:TriggerResultInserter          3.5547e-05    7.80578e-05   0.000173865   7.14935e-05   3.6258e-05       10     
end_path:out1:RootOutput                             8.807e-06    1.84728e-05   5.0474e-05    1.4623e-05    1.1781e-05       10     
end_path:out1:RootOutput(write)                       2.06084       2.90385       4.41858       2.7308       0.616057        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6196.06 MB
  Peak resident set size usage (VmHWM): 3962.59 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.temp.json ====
.:
total 6264264
-rw-r--r-- 1 dune001 lbno      40782 Nov 26 13:19 jobscript.log
-rw-r--r-- 1 dune001 lbno        205 Nov 26 13:19 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno          0 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno          0 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno         31 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno         25 Nov 26 13:19 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno        519 Nov 26 13:19 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno       3574 Nov 26 13:19 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno     204800 Nov 26 13:19 mem.db
-rw-r--r-- 1 dune001 lbno      36864 Nov 26 13:19 time.db
-rw-r--r-- 1 dune001 lbno 6354887967 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252891_8_1_20251126T101242Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno   59406950 Nov 26 13:16 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:45 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno          0 Nov 26 12:42 debugprod.log
-rw-r--r-- 1 dune001 lbno        519 Nov 26 12:42 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:13:01 UTC       justIN version: 01.05.03