justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258730.2@dunegpschedd01.fnal.gov

Jobsub ID258730.2@dunegpschedd01.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 12:32:27
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 16:20:14
From worker nodeHostnamewn124
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 12:33:41
Input fileshd-protodune:pdhd_prod_beam__243946_176_1_20251106T060152Z_gen_g4_IonScintPDExt.root_255581_6_1_20251123T014459Z_PDInt.root
JobscriptExit code0
Real time3h (13328s)
CPU time2h (9813s = 73%)
Max RSS bytes4276412416 (4078 MiB)
Outputting started2025-11-26 16:15:50
Output files
Finished2025-11-26 16:20:14
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

clsFrameSaver: saving 85488 traces tagged "wiener"
FrameSaver: q=2.59896e+06 n=1028646 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7558 XUs and 9579 XVs -> 67 XUVs
C:0 T:1 33743 XUs and 47837 XVs -> 1445 XUVs
C:0 T:2 8903 XUs and 12469 XVs -> 262 XUVs
C:0 T:3 4 XUs and 12 XVs -> 0 XUVs
C:0 T:4 1 XUs and 4 XVs -> 0 XUVs
C:0 T:5 23394 XUs and 12451 XVs -> 647 XUVs
C:0 T:6 18977 XUs and 27152 XVs -> 877 XUVs
C:0 T:7 4 XUs and 3 XVs -> 0 XUVs
3298 XUVs total
2057 collection wire objects
3298 potential space points
Neighbour search...
64138 tests to find 30180 neighbours
Iterating with no regularization...
Begin: 3.60904e+07
0 3.42825e+07
1 3.42073e+07
2 3.42015e+07
Now with regularization...
Begin: 3.21613e+07
0 3.21593e+07
26-Nov-2025 13:15:39 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root"
26-Nov-2025 13:15:39 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            203.137       256.579       360.405       240.755       53.1029        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00275856    0.00983282     0.016686      0.0117856    0.00489712       10     
reco:opdec:Deconvolution                              1.20319       1.4577        1.77211       1.45566       0.15355        10     
reco:ophitspe:OpHitFinderDeco                        0.339618      0.358373      0.383451      0.357197      0.0131828       10     
reco:opflash:OpFlashFinder                           0.118941      0.155205      0.195235      0.157378      0.0262652       10     
reco:opslicer:OpSlicer                               0.378251      0.475581      0.571298      0.469102      0.0735743       10     
reco:wclsdatahd:WireCellToolkit                       74.9606       77.5831       82.7413       77.1456       2.52122        10     
reco:gaushit:GausHitFinder                            1.32222       1.63128       2.0551        1.54163       0.24166        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000451631   0.000815537    0.0016427    0.000666121   0.00035656       10     
reco:reco3d:SpacePointSolver                         0.724731       1.80474       3.51493       1.62112      0.865009        10     
reco:hitpdune:DisambigFromSpacePoints                0.718617       1.42165       2.53108       1.26605      0.551834        10     
reco:pandora:StandardPandora                          28.117        54.1228       98.5708       43.3162       25.7467        10     
reco:pandoraWriter:StandardPandora                   0.531317      0.754794       1.05994      0.719323       0.17879        10     
reco:pandoraTrack:LArPandoraTrackCreation             3.29984       5.15296       11.549        4.16253       2.41054        10     
reco:pandoraShower:LArPandoraShowerCreation           2.14233       4.58644       9.02873       3.72827       2.15105        10     
reco:pandoracalo:Calorimetry                          1.12479       2.54614       7.48837       1.98035       1.71609        10     
reco:pandoracalonosce:Calorimetry                     1.14403       2.3225        6.08562       1.96762       1.34533        10     
reco:pandoraShowercalo:ShowerCalorimetry              2.25137       4.39666       8.6088        3.49047       2.13233        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.37752       4.24246       8.16308       3.51985       1.92538        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0632318     0.124654      0.169144      0.124722      0.0279863       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0665645     0.123302       0.16128      0.123269      0.0277627       10     
reco:emtrkmichelid:EmTrackMichelId                    47.8056       55.8063       68.9895       55.2523       6.7443         10     
reco:crttag:SingleCRTMatchingProducer               0.00899544     0.013519      0.0277557     0.0106618    0.00571562       10     
reco:crtreco:TwoCRTMatchingProducer                  0.0071646     0.0103472     0.0171883    0.00953148    0.00290372       10     
reco:pandora2Track:LArPandoraTrackCreation            7.03139       10.9673       19.7194       9.64994       3.7388         10     
reco:pandora2calo:ShowerCalorimetry                   2.20155       3.46112       5.76435       2.86712       1.23459        10     
reco:pandora2calonosce:ShowerCalorimetry              2.17989       3.39588       6.0651        2.71165       1.25281        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0680658     0.106813      0.146991      0.107133      0.0249199       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.069214      0.107371      0.144068      0.108315      0.0259433       10     
reco:pandora2Shower:LArPandoraShowerCreation          3.21475       4.39745       7.04079       3.91745       1.26986        10     
reco:pandora2Showercalo:ShowerCalorimetry             3.21136       4.50927       7.1571        3.80449       1.34801        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        3.17786       4.62659       7.79891       3.57432       1.69129        10     
[art]:TriggerResults:TriggerResultInserter          3.3412e-05    5.31757e-05   0.000112506   4.5962e-05    2.38517e-05      10     
end_path:out1:RootOutput                             6.181e-06    9.9596e-06    2.9162e-05     8.07e-06     6.49645e-06      10     
end_path:out1:RootOutput(write)                       4.12382       5.67728       6.76773       5.92725      0.856135        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6479.54 MB
  Peak resident set size usage (VmHWM): 4276.41 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.temp.json ====
.:
total 6559656
-rw-r--r-- 1 nobody nobody      49513 Nov 26 13:15 jobscript.log
-rw-r--r-- 1 nobody nobody        203 Nov 26 13:15 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 13:15 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 13:15 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 13:15 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 13:15 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 13:15 time.db
-rw-r--r-- 1 nobody nobody 6661573751 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258730_2_1_20251126T123348Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   55190980 Nov 26 13:14 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:34 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:29 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:29 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:21:39 UTC       justIN version: 01.05.03