justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258764.1@dunegpschedd01.fnal.gov

Jobsub ID258764.1@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 13:20:29
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 16:32:25
From worker nodeHostnamewn66
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 13:21:51
Input fileshd-protodune:pdhd_prod_beam__238442_128_1_20251106T085458Z_gen_g4_IonScintPDExt.root_249506_45_1_20251122T010026Z_PDInt.root
JobscriptExit code0
Real time3h (11164s)
CPU time2h (9228s = 82%)
Max RSS bytes4191510528 (3997 MiB)
Outputting started2025-11-26 16:27:56
Output files
Finished2025-11-26 16:32:25
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1.96449e+06 n=1070350 tag=gauss
wclsFrameSaver: saving 82064 traces tagged "wiener"
FrameSaver: q=2.46969e+06 n=1040022 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 6263 XUs and 6315 XVs -> 66 XUVs
C:0 T:1 60718 XUs and 58092 XVs -> 3076 XUVs
C:0 T:2 23054 XUs and 30233 XVs -> 902 XUVs
C:0 T:3 0 XUs and 4 XVs -> 0 XUVs
C:0 T:5 21056 XUs and 17528 XVs -> 372 XUVs
C:0 T:6 9674 XUs and 13098 XVs -> 433 XUVs
C:0 T:7 6 XUs and 4 XVs -> 0 XUVs
4849 XUVs total
1907 collection wire objects
4849 potential space points
Neighbour search...
253833 tests to find 103206 neighbours
Iterating with no regularization...
Begin: 3.40751e+07
0 3.18743e+07
1 3.17028e+07
2 3.16792e+07
Now with regularization...
Begin: 3.03818e+07
0 3.03738e+07
26-Nov-2025 13:27:44 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root"
26-Nov-2025 13:27:44 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            174.247       244.677       400.446       234.124       61.9215        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00174599     0.033331      0.235391      0.0110764     0.0676113       10     
reco:opdec:Deconvolution                              1.17213       1.45406       1.89027       1.43713      0.203857        10     
reco:ophitspe:OpHitFinderDeco                        0.317938      0.350303      0.376634      0.347699      0.0153681       10     
reco:opflash:OpFlashFinder                           0.0930361     0.143963       0.2063       0.148156      0.034039        10     
reco:opslicer:OpSlicer                               0.345482      0.480153      0.631348      0.465152      0.087881        10     
reco:wclsdatahd:WireCellToolkit                       71.6089       73.9533       76.9199       73.756        1.5614         10     
reco:gaushit:GausHitFinder                            1.31568       1.6695        2.40645       1.6774       0.295441        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000524629   0.000816607   0.00159957    0.000708555   0.000326994      10     
reco:reco3d:SpacePointSolver                         0.779772       1.94583       3.43837       1.63078      0.825783        10     
reco:hitpdune:DisambigFromSpacePoints                0.583119       1.3855        2.55218       1.46856      0.559414        10     
reco:pandora:StandardPandora                          18.3926       47.1642       111.13        43.8354       24.7654        10     
reco:pandoraWriter:StandardPandora                   0.524239      0.782498       1.61668      0.666697      0.317562        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.88399       4.49794       8.21153       3.88175       2.14823        10     
reco:pandoraShower:LArPandoraShowerCreation           1.65783       4.57646       8.58314       3.76148       2.17944        10     
reco:pandoracalo:Calorimetry                          1.05635       2.18793       3.74416       2.02101      0.903546        10     
reco:pandoracalonosce:Calorimetry                     1.01595       2.16644       3.94488       1.98909      0.934231        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.78995       4.46172       8.31938       4.46709       2.03753        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.69549       4.17517       7.88847       4.19243       1.90521        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0790737     0.127079      0.169609      0.127139      0.0245591       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0823565     0.124927      0.167305       0.12335      0.0219166       10     
reco:emtrkmichelid:EmTrackMichelId                    41.0747       55.4105       81.4713       53.7703       10.8504        10     
reco:crttag:SingleCRTMatchingProducer               0.00827308     0.0134942     0.0263733     0.0121918    0.00514627       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00616418    0.00958377     0.016654     0.00839504    0.00340293       10     
reco:pandora2Track:LArPandoraTrackCreation            4.36138       10.9303       25.305        10.1387       5.46898        10     
reco:pandora2calo:ShowerCalorimetry                   1.49382       3.50282       7.83766       3.44831       1.73109        10     
reco:pandora2calonosce:ShowerCalorimetry              1.41619       3.43639       7.53301       3.25348       1.64202        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0749318     0.115082      0.151365      0.113213      0.0192847       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0799308     0.114479      0.143879      0.116207      0.0173002       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.65323       4.35978       9.27554       4.10368       1.98726        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.84336       4.6181        10.0781       4.44573       2.15622        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.81795       4.44707       10.2837       4.07859       2.23244        10     
[art]:TriggerResults:TriggerResultInserter          2.9951e-05    5.4915e-05    0.000162276   4.1116e-05    3.68232e-05      10     
end_path:out1:RootOutput                             5.75e-06     1.10491e-05   3.7061e-05     7.205e-06    9.28306e-06      10     
end_path:out1:RootOutput(write)                       4.55064       5.80657       6.68952       5.95951      0.595884        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6578.05 MB
  Peak resident set size usage (VmHWM): 4124.86 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.temp.json ====
.:
total 6291328
-rw-r--r-- 1 nobody nobody      41557 Nov 26 13:27 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 13:27 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 13:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 13:27 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 13:27 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 13:27 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 13:27 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 13:27 time.db
-rw-r--r-- 1 nobody nobody 6387732783 Nov 26 13:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258764_1_1_20251126T132159Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   54271380 Nov 26 13:26 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:50 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:43 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:43 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:12:59 UTC       justIN version: 01.05.03