justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258765.4@dunegpschedd01.fnal.gov

Jobsub ID258765.4@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 13:22:29
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 16:38:45
From worker nodeHostnamewn78
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 13:23:55
Input fileshd-protodune:pdhd_prod_beam__238170_54_1_20251106T033908Z_gen_g4_IonScintPDExt.root_249919_132_1_20251123T024023Z_PDInt.root
JobscriptExit code0
Real time3h (11428s)
CPU time2h (9485s = 82%)
Max RSS bytes4214951936 (4019 MiB)
Outputting started2025-11-26 16:34:25
Output files
Finished2025-11-26 16:38:45
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

"wiener"
FrameSaver: q=2.63543e+06 n=1197562 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 6215 XUs and 6401 XVs -> 43 XUVs
C:0 T:1 25979 XUs and 23824 XVs -> 576 XUVs
C:0 T:2 33352 XUs and 22596 XVs -> 465 XUVs
C:0 T:3 9 XUs and 8 XVs -> 0 XUVs
C:0 T:5 26273 XUs and 27599 XVs -> 810 XUVs
C:0 T:6 37542 XUs and 37464 XVs -> 1162 XUVs
C:0 T:7 0 XUs and 2 XVs -> 0 XUVs
3056 XUVs total
2046 collection wire objects
3056 potential space points
Neighbour search...
38966 tests to find 20376 neighbours
Iterating with no regularization...
Begin: 2.6227e+07
0 2.50579e+07
1 2.50193e+07
2 2.5016e+07
Now with regularization...
Begin: 2.35738e+07
0 2.3572e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 13:34:15 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root"
26-Nov-2025 13:34:15 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            193.345       247.309       320.236        248.6        41.8005        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00191339     0.0110062     0.017743      0.0134809    0.00521778       10     
reco:opdec:Deconvolution                              1.22397       1.42358       1.64842       1.42394      0.128895        10     
reco:ophitspe:OpHitFinderDeco                        0.262354      0.338463      0.362292      0.350536      0.0282312       10     
reco:opflash:OpFlashFinder                            0.10875      0.149596      0.220214      0.147689      0.0294468       10     
reco:opslicer:OpSlicer                               0.304914      0.454237      0.597632       0.45071      0.0872188       10     
reco:wclsdatahd:WireCellToolkit                       66.221        70.8441       73.4544       70.9883       2.03635        10     
reco:gaushit:GausHitFinder                           0.862834       1.62532       2.04039       1.70325      0.334982        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000416289   0.000719856   0.00164147    0.000638969   0.00032368       10     
reco:reco3d:SpacePointSolver                         0.954679       1.7921        3.60345       1.59023       0.74987        10     
reco:hitpdune:DisambigFromSpacePoints                 1.0201        1.60417       2.29713       1.53171      0.414019        10     
reco:pandora:StandardPandora                          24.5493       53.7491       98.4167       46.8608       21.8264        10     
reco:pandoraWriter:StandardPandora                   0.500473      0.792657       1.41696      0.755879      0.255029        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.67074       3.84336       5.82662       3.75407       1.09116        10     
reco:pandoraShower:LArPandoraShowerCreation           1.9222        4.04396       6.51155       3.62427        1.69          10     
reco:pandoracalo:Calorimetry                          1.3799        2.05585       3.31607       1.81705      0.619071        10     
reco:pandoracalonosce:Calorimetry                     1.39264       1.92387       2.84793       1.79872      0.433021        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.11526       3.75192       6.48432       3.41327       1.43103        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.99684       3.79956       6.79702       3.30002       1.59036        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0650137      0.12092      0.159171      0.119498      0.024905        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0691108     0.119225      0.161336      0.116001      0.0243241       10     
reco:emtrkmichelid:EmTrackMichelId                    46.6643       57.0495       69.2761        54.67        6.86723        10     
reco:crttag:SingleCRTMatchingProducer               0.00688393     0.0105543     0.0132779     0.0110265    0.00199838       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00589351    0.00921321     0.011932      0.0090591    0.00202547       10     
reco:pandora2Track:LArPandoraTrackCreation            5.59292       11.223        18.6851       10.0915       4.27084        10     
reco:pandora2calo:ShowerCalorimetry                   1.56614       3.59818       6.57241       3.07982       1.55467        10     
reco:pandora2calonosce:ShowerCalorimetry              1.80346       3.39373       5.36468       3.03722       1.29157        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0567294     0.116351      0.148485      0.120855      0.0236667       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0561445     0.117373      0.145159       0.1229       0.0242846       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.11506       4.49277       6.9119        4.24413       1.62067        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.51506       4.63463       7.43967       4.04069       1.65492        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.25501       4.34596       6.91714       3.77746       1.50937        10     
[art]:TriggerResults:TriggerResultInserter          2.6311e-05    7.17104e-05   0.000376367   3.6952e-05    0.000101889      10     
end_path:out1:RootOutput                             5.35e-06     1.39966e-05   5.5573e-05    6.4655e-06    1.61839e-05      10     
end_path:out1:RootOutput(write)                       3.70177       5.67094       7.33034       6.05706       1.11021        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6089.47 MB
  Peak resident set size usage (VmHWM): 3863.91 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.temp.json ====
.:
total 6216868
-rw-r--r-- 1 nobody nobody      40959 Nov 26 13:34 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 13:34 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 13:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 13:34 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 13:34 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 13:34 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     200704 Nov 26 13:34 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 13:34 time.db
-rw-r--r-- 1 nobody nobody 6306851424 Nov 26 13:34 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258765_4_1_20251126T132404Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   58912830 Nov 26 13:32 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:54 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:50 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:49 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:21:45 UTC       justIN version: 01.05.03