justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253026.3@dunegpschedd02.fnal.gov

Jobsub ID253026.3@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 12:06:26
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 15:05:45
From worker nodeHostnamewn83
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 12:07:43
Input fileshd-protodune:pdhd_prod_beam__239170_4_1_20251107T000117Z_gen_g4_IonScintPDExt.root_249561_97_1_20251122T081233Z_PDInt.root
JobscriptExit code0
Real time2h (10416s)
CPU time2h (8712s = 83%)
Max RSS bytes4220563456 (4025 MiB)
Outputting started2025-11-26 15:01:21
Output files
Finished2025-11-26 15:05:45
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

s tagged "gauss"
FrameSaver: q=2.02936e+06 n=1085290 tag=gauss
wclsFrameSaver: saving 80272 traces tagged "wiener"
FrameSaver: q=2.56995e+06 n=1074862 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12484 XUs and 9077 XVs -> 173 XUVs
C:0 T:1 75840 XUs and 57171 XVs -> 3340 XUVs
C:0 T:2 14202 XUs and 18253 XVs -> 294 XUVs
C:0 T:4 190 XUs and 90 XVs -> 1 XUVs
C:0 T:5 56129 XUs and 60890 XVs -> 1688 XUVs
C:0 T:6 2312 XUs and 2828 XVs -> 147 XUVs
5643 XUVs total
2455 collection wire objects
5643 potential space points
Neighbour search...
303699 tests to find 98540 neighbours
Iterating with no regularization...
Begin: 1.04758e+08
0 9.47456e+07
1 9.44693e+07
2 9.4448e+07
Now with regularization...
Begin: 9.31167e+07
0 9.31124e+07
26-Nov-2025 12:01:11 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root"
26-Nov-2025 12:01:11 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            161.757       230.697       292.22        235.93        40.5981        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                               0.0016156    0.00904165     0.0165684    0.00997288    0.00469162       10     
reco:opdec:Deconvolution                              1.07021       1.33488       1.64277       1.34591       0.17109        10     
reco:ophitspe:OpHitFinderDeco                        0.179228      0.322931      0.364349      0.336363      0.0494801       10     
reco:opflash:OpFlashFinder                           0.0782209      0.13287      0.178946      0.140415      0.0310184       10     
reco:opslicer:OpSlicer                               0.170832      0.417669      0.592225      0.441213       0.11079        10     
reco:wclsdatahd:WireCellToolkit                       69.689        71.1496        72.53        71.4895       1.03056        10     
reco:gaushit:GausHitFinder                           0.817398       1.52914       2.19857       1.50563      0.322425        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00049685    0.000634665   0.00103854    0.000578722   0.000150339      10     
reco:reco3d:SpacePointSolver                         0.449155       2.04747       3.5473        2.19744      0.861647        10     
reco:hitpdune:DisambigFromSpacePoints                 0.4189        1.29862       1.86359       1.26319      0.418993        10     
reco:pandora:StandardPandora                          18.0357       52.2854       82.8366       52.2259       22.2428        10     
reco:pandoraWriter:StandardPandora                   0.501793      0.792256       1.50234      0.712985      0.261745        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.46331       3.31728       6.33014       3.47988       1.41726        10     
reco:pandoraShower:LArPandoraShowerCreation           1.65204       3.22411       5.47633       3.16509       1.10541        10     
reco:pandoracalo:Calorimetry                          0.92329       1.62389       2.71661       1.54783      0.560802        10     
reco:pandoracalonosce:Calorimetry                    0.886299       1.70302       2.82303       1.59456      0.627517        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.65585       3.04728       5.34019       2.84838       1.06495        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.41503       2.97055       5.0911        3.05946       1.03936        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0560594     0.105885      0.135209      0.108143      0.0214596       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.059307      0.108776      0.143785      0.111456      0.0214131       10     
reco:emtrkmichelid:EmTrackMichelId                    43.2517       51.5473       62.6815       50.2082       6.88098        10     
reco:crttag:SingleCRTMatchingProducer               0.00838854     0.0125781     0.0221186     0.0113076    0.00438484       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00650415     0.0096779     0.0167521    0.00907838    0.00299412       10     
reco:pandora2Track:LArPandoraTrackCreation            4.2265        8.57957       12.9512       7.85737       2.92487        10     
reco:pandora2calo:ShowerCalorimetry                   1.54739       2.88807       4.8161        2.67839      0.939514        10     
reco:pandora2calonosce:ShowerCalorimetry              1.62148       2.76521        4.785        2.59643       0.87571        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0938167     0.111692       0.13676       0.10875      0.0123508       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0945515     0.111121      0.129321      0.108165      0.0110664       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.73975       4.06664       8.92318       3.44635       1.98046        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.88725       3.87026       7.2477        3.35204       1.47927        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.91752       3.82459       7.38704       3.39123       1.4842         10     
[art]:TriggerResults:TriggerResultInserter          3.1611e-05    5.52021e-05   0.000117744   4.5552e-05    2.59392e-05      10     
end_path:out1:RootOutput                             5.74e-06     1.02035e-05   3.6561e-05    7.1205e-06    8.85204e-06      10     
end_path:out1:RootOutput(write)                       3.86163       5.28583       6.66827       5.25738      0.823085        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5943.38 MB
  Peak resident set size usage (VmHWM): 3765.49 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.temp.json ====
.:
total 5722728
-rw-r--r-- 1 nobody nobody      46506 Nov 26 12:01 jobscript.log
-rw-r--r-- 1 nobody nobody        202 Nov 26 12:01 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 12:01 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:01 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 12:01 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 12:01 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 12:01 time.db
-rw-r--r-- 1 nobody nobody 5807062654 Nov 26 12:01 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_3_1_20251126T120750Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   52691450 Nov 26 11:59 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:24 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:20 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:19 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 10:31:52 UTC       justIN version: 01.06.00