justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258721.2@dunegpschedd01.fnal.gov

Jobsub ID258721.2@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 12:18:26
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 18:01:58
From worker nodeHostnamewn121.cat.cbpf.br
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 12:20:05
Input fileshd-protodune:pdhd_prod_beam__244248_113_1_20251106T103956Z_gen_g4_IonScintPDExt.root_255411_152_1_20251122T173844Z_PDInt.root
JobscriptExit code0
Real time5h (20215s)
CPU time3h (13258s = 65%)
Max RSS bytes4195311616 (4000 MiB)
Outputting started2025-11-26 17:57:01
Output files
Finished2025-11-26 18:01:58
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1 XUs and 63739 XVs -> 1288 XUVs
C:0 T:2 58553 XUs and 69460 XVs -> 1377 XUVs
C:0 T:3 6 XUs and 2 XVs -> 0 XUVs
C:0 T:4 17 XUs and 11 XVs -> 0 XUVs
C:0 T:5 30113 XUs and 31924 XVs -> 563 XUVs
C:0 T:6 29864 XUs and 44711 XVs -> 900 XUVs
4221 XUVs total
2684 collection wire objects
4221 potential space points
Neighbour search...
61321 tests to find 29888 neighbours
Iterating with no regularization...
Begin: 4.53614e+07
0 4.29891e+07
1 4.28731e+07
2 4.28678e+07
Now with regularization...
Begin: 4.14475e+07
0 4.14456e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 14:56:32 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root"
26-Nov-2025 14:56:32 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            342.076       455.108       676.572       420.316       106.149        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00442935     0.0165571     0.0312905     0.0151357    0.00728963       10     
reco:opdec:Deconvolution                              0.86624       1.51276       2.85038       1.38728      0.537594        10     
reco:ophitspe:OpHitFinderDeco                        0.241766      0.636938       1.39665      0.513495      0.373114        10     
reco:opflash:OpFlashFinder                           0.0683647      0.17567      0.384956      0.156222      0.0852903       10     
reco:opslicer:OpSlicer                               0.220841      0.461586       1.2266       0.401093       0.27181        10     
reco:wclsdatahd:WireCellToolkit                       93.4123       141.583       222.409       135.821       41.1459        10     
reco:gaushit:GausHitFinder                            1.05148       1.84089       3.38017       1.76105      0.594481        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00266132    0.00465332    0.00821335    0.00440282    0.00151701       10     
reco:reco3d:SpacePointSolver                          1.60879       2.64585       4.61322       2.45231      0.903788        10     
reco:hitpdune:DisambigFromSpacePoints                0.853783       1.32655       1.84172       1.30244      0.297842        10     
reco:pandora:StandardPandora                          28.5638       55.1781       67.081        60.4614       12.6172        10     
reco:pandoraWriter:StandardPandora                   0.636591       1.05522       1.57658       1.02028      0.263627        10     
reco:pandoraTrack:LArPandoraTrackCreation             3.74075       13.4985       27.6491       11.5744       8.68963        10     
reco:pandoraShower:LArPandoraShowerCreation           2.81574       14.5541       31.7809       8.98026       11.7375        10     
reco:pandoracalo:Calorimetry                          1.66915       5.61428       11.4771       3.36111       3.91883        10     
reco:pandoracalonosce:Calorimetry                     1.30296       4.8034        9.58123       4.08701       3.09446        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.67818       9.6122        21.5157       6.69699       7.19848        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.47866       8.94528       22.5545       6.91352       6.83152        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0802282     0.141049      0.228936      0.134776      0.0447404       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0779652      0.14546      0.239989      0.133646      0.0508524       10     
reco:emtrkmichelid:EmTrackMichelId                    88.0385       110.39         153.9        104.664       20.1319        10     
reco:crttag:SingleCRTMatchingProducer                0.0215266     0.0409999     0.0896743     0.0340826     0.0196446       10     
reco:crtreco:TwoCRTMatchingProducer                  0.0116228     0.0324935     0.0543327     0.0335512     0.0118403       10     
reco:pandora2Track:LArPandoraTrackCreation            9.94496       30.2719       79.4913       25.9322       19.1044        10     
reco:pandora2calo:ShowerCalorimetry                   2.19267       7.28833       14.6133       5.20336       4.4656         10     
reco:pandora2calonosce:ShowerCalorimetry              2.06279       7.33636       20.7022       5.56142       5.42636        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0827099     0.128074      0.171819      0.123883      0.0275155       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0863972     0.123512       0.15885      0.118315      0.0218932       10     
reco:pandora2Shower:LArPandoraShowerCreation          4.11467       11.8162       19.4287       9.50826       5.45826        10     
reco:pandora2Showercalo:ShowerCalorimetry              3.159        9.21684       18.5499       8.16104       4.83987        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.67588       9.46975       19.6087       7.16809       6.36917        10     
[art]:TriggerResults:TriggerResultInserter          5.4912e-05    0.000102897   0.000438057   6.87375e-05   0.000111952      10     
end_path:out1:RootOutput                             8.09e-06     1.86978e-05   6.9883e-05    1.3306e-05    1.72449e-05      10     
end_path:out1:RootOutput(write)                       2.79753       4.75608       8.66361       4.49072       1.54983        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6373.06 MB
  Peak resident set size usage (VmHWM): 4140.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.temp.json ====
.:
total 6352536
-rw-r--r-- 1 nobody nobody      41514 Nov 26 14:57 jobscript.log
-rw-r--r-- 1 nobody nobody        205 Nov 26 14:57 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:57 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 14:57 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 14:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 14:56 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 14:56 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 14:56 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 14:56 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 14:56 time.db
-rw-r--r-- 1 nobody nobody 6451287573 Nov 26 14:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258721_2_1_20251126T122013Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   53392590 Nov 26 14:49 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:41 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:34 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 13:33 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 09:19:34 UTC       justIN version: 01.05.03