justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253026.6@dunegpschedd02.fnal.gov

Jobsub ID253026.6@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 12:06:26
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce04
Last heartbeat2025-11-26 16:24:29
From worker nodeHostnamewn123
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 12:07:35
Input fileshd-protodune:pdhd_prod_beam__243946_67_1_20251106T055937Z_gen_g4_IonScintPDExt.root_255576_105_1_20251123T013557Z_PDInt.root
JobscriptExit code0
Real time4h (15142s)
CPU time2h (10123s = 66%)
Max RSS bytes4531064832 (4321 MiB)
Outputting started2025-11-26 16:19:59
Output files
Finished2025-11-26 16:24:29
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

: q=2.02738e+06 n=981868 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 5131 XUs and 5280 XVs -> 24 XUVs
C:0 T:1 15111 XUs and 16966 XVs -> 170 XUVs
C:0 T:2 15338 XUs and 8606 XVs -> 147 XUVs
C:0 T:4 9 XUs and 7 XVs -> 3 XUVs
C:0 T:5 19809 XUs and 19066 XVs -> 657 XUVs
C:0 T:6 10944 XUs and 16599 XVs -> 838 XUVs
C:0 T:7 9 XUs and 6 XVs -> 0 XUVs
1839 XUVs total
1186 collection wire objects
1839 potential space points
Neighbour search...
27189 tests to find 16282 neighbours
Iterating with no regularization...
Begin: 1.48238e+07
0 1.43182e+07
1 1.43019e+07
2 1.43005e+07
Now with regularization...
Begin: 1.32164e+07
0 1.32158e+07
TestBeamParticleCreationAlgorithm::SetupTestBeamVertex - Test beam particle has no initial vertex
26-Nov-2025 13:19:46 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root"
26-Nov-2025 13:19:46 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            184.281       285.943       432.997       273.118       73.1413        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00223872     0.0113988     0.0235848     0.0112327    0.00620811       10     
reco:opdec:Deconvolution                              1.15947       1.36957       1.69469       1.30472      0.166979        10     
reco:ophitspe:OpHitFinderDeco                        0.192669      0.325465      0.407605      0.347468      0.0692125       10     
reco:opflash:OpFlashFinder                           0.0965677     0.131491      0.158235      0.134437      0.0176852       10     
reco:opslicer:OpSlicer                               0.356316      0.440613      0.527766      0.445233      0.0631991       10     
reco:wclsdatahd:WireCellToolkit                       79.0391       85.2556       92.222        84.9399       3.59154        10     
reco:gaushit:GausHitFinder                            1.27093       1.66486       2.19351       1.67577      0.306738        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.00104709    0.00170886    0.00250994    0.00163845    0.000488865      10     
reco:reco3d:SpacePointSolver                         0.577171       2.24811        4.696        1.8393        1.16629        10     
reco:hitpdune:DisambigFromSpacePoints                0.541401       1.3632        2.37932       1.26738      0.545695        10     
reco:pandora:StandardPandora                          17.1401       51.6287       82.7482       48.0337       22.5916        10     
reco:pandoraWriter:StandardPandora                   0.448168      0.889126       1.24158      0.885904      0.248567        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.30795       6.9272        10.9916       7.51694       2.54891        10     
reco:pandoraShower:LArPandoraShowerCreation           1.73451       7.26372       13.6642       7.40164       3.48575        10     
reco:pandoracalo:Calorimetry                         0.732096       3.51823       6.78119       3.52722       1.80313        10     
reco:pandoracalonosce:Calorimetry                    0.683152       3.15878       5.64967        3.161        1.43081        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.43518       5.68882       10.7125       5.78664       2.75864        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.32448       4.8777        7.43294       4.76596       2.07193        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0630893     0.130708      0.166667       0.14106      0.0312963       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry       0.08794      0.140688        0.177       0.150539      0.0286052       10     
reco:emtrkmichelid:EmTrackMichelId                    42.5486       60.9877       83.3051       61.1747       11.524         10     
reco:crttag:SingleCRTMatchingProducer                0.0102944     0.0177301     0.0367518     0.0139465     0.0087076       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00654024     0.0150512     0.0346835    0.00890361     0.0109571       10     
reco:pandora2Track:LArPandoraTrackCreation            4.65354       14.6141       28.4326       15.5739       7.51819        10     
reco:pandora2calo:ShowerCalorimetry                   1.72142       4.57322       16.3308       2.92035       4.14067        10     
reco:pandora2calonosce:ShowerCalorimetry              1.44318       4.61571       14.5389       3.50636       3.60284        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0838262     0.126116      0.148058      0.137291      0.0220594       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0868121     0.127336      0.155361      0.127287      0.0238011       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.08248       6.34293       16.3802       5.48369       4.18249        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.18572       5.93684       20.9459       4.44958       5.21058        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.08597       5.69532       19.8545       4.42332       4.93319        10     
[art]:TriggerResults:TriggerResultInserter          4.7932e-05    7.50774e-05   0.00020435    5.95625e-05   4.43789e-05      10     
end_path:out1:RootOutput                             6.781e-06    1.51358e-05   6.0743e-05    1.04205e-05   1.53228e-05      10     
end_path:out1:RootOutput(write)                       3.77185       5.49913       7.6525        5.29207       1.02739        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6029.43 MB
  Peak resident set size usage (VmHWM): 3812.24 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.temp.json ====
.:
total 5674788
-rw-r--r-- 1 nobody nobody      40287 Nov 26 13:19 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 13:19 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 13:19 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 13:19 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 13:19 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 13:19 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 13:19 time.db
-rw-r--r-- 1 nobody nobody 5758949217 Nov 26 13:19 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_253026_6_1_20251126T120742Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   51716190 Nov 26 13:18 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:34 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:28 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:28 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:21:43 UTC       justIN version: 01.05.03