justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258531.9@dunegpschedd01.fnal.gov

Jobsub ID258531.9@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 09:47:19
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 13:57:53
From worker nodeHostnamewn113
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 09:48:57
Input fileshd-protodune:pdhd_prod_beam__238227_88_1_20251106T052052Z_gen_g4_IonScintPDExt.root_249896_50_1_20251123T000849Z_PDInt.root
JobscriptExit code0
Real time4h (14662s)
CPU time3h (11422s = 77%)
Max RSS bytes4167303168 (3974 MiB)
Outputting started2025-11-26 13:53:21
Output files
Finished2025-11-26 13:57:53
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ged "gauss"
FrameSaver: q=2.45373e+06 n=1224526 tag=gauss
wclsFrameSaver: saving 101717 traces tagged "wiener"
FrameSaver: q=3.14243e+06 n=1169427 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11053 XUs and 10648 XVs -> 81 XUVs
C:0 T:1 64594 XUs and 76605 XVs -> 2359 XUVs
C:0 T:2 67212 XUs and 66727 XVs -> 2152 XUVs
C:0 T:3 15 XUs and 40 XVs -> 0 XUVs
C:0 T:5 18584 XUs and 17093 XVs -> 257 XUVs
C:0 T:6 60317 XUs and 75536 XVs -> 1946 XUVs
6795 XUVs total
3037 collection wire objects
6795 potential space points
Neighbour search...
212731 tests to find 114610 neighbours
Iterating with no regularization...
Begin: 4.04434e+07
0 3.71528e+07
1 3.68998e+07
2 3.68709e+07
Now with regularization...
Begin: 3.53288e+07
0 3.53198e+07
26-Nov-2025 10:53:09 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root"
26-Nov-2025 10:53:09 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            262.179       402.492       598.568       381.89        109.721        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00341008     0.0139563     0.0229915     0.0136552    0.00605248       10     
reco:opdec:Deconvolution                             0.968273       1.45258       2.04743       1.48541      0.327509        10     
reco:ophitspe:OpHitFinderDeco                        0.219612      0.450805      0.773546       0.42061      0.174483        10     
reco:opflash:OpFlashFinder                           0.0915215     0.154769      0.232795      0.152973      0.0474847       10     
reco:opslicer:OpSlicer                               0.276212      0.436365      0.593212      0.416025      0.116409        10     
reco:wclsdatahd:WireCellToolkit                       84.3783       109.391       148.13        106.197       20.2113        10     
reco:gaushit:GausHitFinder                           0.893982       1.6027        2.19886       1.65241      0.406428        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00114152    0.00334545    0.00920963    0.00277317    0.00217672       10     
reco:reco3d:SpacePointSolver                          1.1623        2.79447       5.64361       2.44569       1.47146        10     
reco:hitpdune:DisambigFromSpacePoints                0.715681       1.71527       3.27245       1.42446       0.86476        10     
reco:pandora:StandardPandora                          37.6563       61.5529       140.868       43.7377       32.4165        10     
reco:pandoraWriter:StandardPandora                   0.689531      0.914169       1.30676      0.853356      0.191484        10     
reco:pandoraTrack:LArPandoraTrackCreation             4.55379       10.5228       19.0593       10.671        3.63649        10     
reco:pandoraShower:LArPandoraShowerCreation           3.64712       11.5067       22.9998       10.7397       5.39676        10     
reco:pandoracalo:Calorimetry                          1.93319       4.6497        8.98853       4.04675       1.83273        10     
reco:pandoracalonosce:Calorimetry                     1.49184       4.47141       8.06635       4.39382       1.66538        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.4744        11.7542       34.1993       8.92027       9.0627         10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.37391       9.47582       20.2533       7.8456        5.63274        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0576236     0.141593      0.194799      0.144557      0.0433763       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0602516     0.139995      0.205237      0.144794      0.0414028       10     
reco:emtrkmichelid:EmTrackMichelId                    56.0043       81.685        114.146       79.5118       17.8115        10     
reco:crttag:SingleCRTMatchingProducer                0.015276      0.0308766     0.0649745     0.0257453     0.0143118       10     
reco:crtreco:TwoCRTMatchingProducer                  0.0126682     0.0313208     0.0725693     0.0266577     0.0171604       10     
reco:pandora2Track:LArPandoraTrackCreation            12.2884       28.774        47.5818       26.0199       10.7903        10     
reco:pandora2calo:ShowerCalorimetry                   3.04234       8.9357        20.6416       7.22487       4.85207        10     
reco:pandora2calonosce:ShowerCalorimetry              2.86157       8.78857       23.3289       6.77752       5.58887        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.104001      0.146979      0.212937       0.14726      0.0305988       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.102529      0.147754      0.223011      0.146928      0.0325034       10     
reco:pandora2Shower:LArPandoraShowerCreation          4.30414       13.2893        27.47        13.2897       6.44565        10     
reco:pandora2Showercalo:ShowerCalorimetry             4.01085       11.3994       27.8134       9.8501        6.83579        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        3.64888       10.9324       23.1508       9.41273       5.79333        10     
[art]:TriggerResults:TriggerResultInserter          4.4622e-05    9.2976e-05    0.000232008   7.3557e-05    4.9778e-05       10     
end_path:out1:RootOutput                              8.8e-06     1.67758e-05   5.2682e-05    1.2871e-05    1.23258e-05      10     
end_path:out1:RootOutput(write)                       3.47236       4.7155        5.44712       4.83918       0.63083        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6332.22 MB
  Peak resident set size usage (VmHWM): 4105.61 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.temp.json ====
.:
total 6392372
-rw-r--r-- 1 nobody nobody      41149 Nov 26 10:53 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 10:53 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 10:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 10:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 10:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 10:53 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 10:53 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 10:53 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 10:53 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 10:53 time.db
-rw-r--r-- 1 nobody nobody 6486825347 Nov 26 10:53 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258531_9_1_20251126T094905Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   58644540 Nov 26 10:49 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:50 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 09:41 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 09:41 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:20:23 UTC       justIN version: 01.06.00