justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261437.106@dunegpschedd02.fnal.gov

Jobsub ID261437.106@dunegpschedd02.fnal.gov
Workflow ID11161
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-12 01:34:54
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-12-12 08:57:18
From worker nodeHostnamee4023
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-12 07:36:06
Input fileshd-protodune:pdhd_prod_beam__249073_109_1_20251113T221556Z_gen_g4_IonScintPDExt.root_248478_130_1_20251119T160113Z_PDInt.root
JobscriptExit code0
Real time1h (4848s)
CPU time1h (4413s = 91%)
Max RSS bytes7087284224 (6758 MiB)
Outputting started2025-12-12 08:56:55
Output files
Finished2025-12-12 08:57:18
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

UVs
C:0 T:1 50594 XUs and 51022 XVs -> 1018 XUVs
C:0 T:2 14762 XUs and 13874 XVs -> 609 XUVs
C:0 T:3 9 XUs and 14 XVs -> 0 XUVs
C:0 T:4 0 XUs and 7 XVs -> 0 XUVs
C:0 T:5 32112 XUs and 34238 XVs -> 1271 XUVs
C:0 T:6 17030 XUs and 13483 XVs -> 309 XUVs
3296 XUVs total
2059 collection wire objects
3296 potential space points
Neighbour search...
64088 tests to find 35212 neighbours
Iterating with no regularization...
Begin: 4.54958e+07
0 4.35792e+07
1 4.3473e+07
2 4.34613e+07
Now with regularization...
Begin: 4.12639e+07
0 4.12602e+07
Begin processing the 10th record. run: 20250613 subRun: 1 event: 429 at 12-Dec-2025 02:54:05 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 18730!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 91241 traces, 91241 summary
Retagger: tagging trace set: gauss with 75426 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75426 traces tagged "gauss"
FrameSaver: q=2.40739e+06 n=1277233 tag=gauss
wclsFrameSaver: saving 91241 traces tagged "wiener"
FrameSaver: q=2.7725e+06 n=1230641 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8599 XUs and 9528 XVs -> 116 XUVs
C:0 T:1 42795 XUs and 37947 XVs -> 1176 XUVs
C:0 T:2 14942 XUs and 18903 XVs -> 1564 XUVs
C:0 T:3 17 XUs and 25 XVs -> 4 XUVs
C:0 T:5 25074 XUs and 27383 XVs -> 885 XUVs
C:0 T:6 19402 XUs and 18471 XVs -> 745 XUVs
C:0 T:7 9 XUs and 9 XVs -> 0 XUVs
4490 XUVs total
2369 collection wire objects
4490 potential space points
Neighbour search...
133236 tests to find 77506 neighbours
Iterating with no regularization...
Begin: 2.7777e+07
0 2.65644e+07
1 2.64894e+07
2 2.64793e+07
Now with regularization...
Begin: 2.48878e+07
0 2.48845e+07
12-Dec-2025 02:56:46 CST  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root"
12-Dec-2025 02:56:46 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            98.2837       133.063       162.474       140.184       22.2265        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000748264   0.00514896    0.00898383    0.00492223    0.00261214       10     
reco:opdec:Deconvolution                             0.713486      0.959333       1.23065      0.930864      0.157774        10     
reco:ophitspe:OpHitFinderDeco                        0.140471      0.220446      0.275432      0.246089      0.0528851       10     
reco:opflash:OpFlashFinder                           0.0486736     0.103295      0.151006      0.106945      0.0317649       10     
reco:opslicer:OpSlicer                               0.120309       0.27596      0.428653      0.275555      0.0937474       10     
reco:wclsdatahd:WireCellToolkit                       44.9224       50.7844       58.2321       50.8074       3.7413         10     
reco:gaushit:GausHitFinder                           0.418593      0.763401       1.13252      0.714138       0.21914        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000243226   0.000403052   0.00083104    0.000316774   0.000183437      10     
reco:reco3d:SpacePointSolver                         0.242853      0.557024      0.920796      0.561366      0.231745        10     
reco:hitpdune:DisambigFromSpacePoints                0.192182      0.593492       0.95784      0.629041      0.246661        10     
reco:pandora:StandardPandora                          8.82015       15.6867       23.2149       15.6637       4.92419        10     
reco:pandoraWriter:StandardPandora                   0.178023       0.30334      0.464075      0.315001      0.0833365       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.633481       1.34477       1.92396       1.46093      0.462838        10     
reco:pandoraShower:LArPandoraShowerCreation          0.722872       1.58448       2.49348       1.81341      0.607516        10     
reco:pandoracalo:Calorimetry                         0.446262      0.781816       1.10616      0.797136      0.233844        10     
reco:pandoracalonosce:Calorimetry                    0.440688      0.763515       1.07511      0.782356      0.228989        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.754229       1.59001       2.82638       1.76004      0.642322        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.656027       1.50203       2.63949       1.59224      0.621216        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0306517     0.0523944     0.0773491     0.0529569     0.0164905       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0309328     0.0514686     0.0757741     0.0495151     0.0162223       10     
reco:emtrkmichelid:EmTrackMichelId                    23.7865       39.1797       51.1047       42.0984       8.49916        10     
reco:crttag:SingleCRTMatchingProducer               0.00488146    0.00680067    0.00983084    0.00672119    0.00149836       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00405484    0.00541641    0.00808416    0.00546633    0.00113618       10     
reco:pandora2Track:LArPandoraTrackCreation            1.80697       4.40215       8.23627       4.63524        1.696         10     
reco:pandora2calo:ShowerCalorimetry                  0.815571       1.53049       3.08051       1.55073      0.640383        10     
reco:pandora2calonosce:ShowerCalorimetry             0.755227       1.47486       3.06034       1.44919      0.638035        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0399496     0.0767155     0.118908      0.0768609     0.0229411       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.035533      0.0754219      0.11829      0.0753841     0.0236794       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.726207       1.78817       3.47045       1.92691      0.745037        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.714433       1.88353       3.85382       1.98358       0.82624        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.691982       1.85569       3.73036        1.993       0.833617        10     
[art]:TriggerResults:TriggerResultInserter          2.7141e-05    5.29716e-05   0.000171412   3.8753e-05    4.13315e-05      10     
end_path:out1:RootOutput                             7.454e-06    1.34705e-05   5.0775e-05     9.413e-06    1.2555e-05       10     
end_path:out1:RootOutput(write)                       1.52026       2.78207       3.66419       2.97766      0.715497        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6359.58 MB
  Peak resident set size usage (VmHWM): 4086.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.ext.json ====
Error, specified input file does not exist.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.temp.json ====
.:
total 6224040
-rw-r--r-- 1 slot1_33 slot1_33      39386 Dec 12 02:56 jobscript.log
-rw-r--r-- 1 slot1_33 slot1_33        199 Dec 12 02:56 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_33 slot1_33          0 Dec 12 02:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.json
-rw-r--r-- 1 slot1_33 slot1_33          0 Dec 12 02:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 slot1_33 slot1_33         44 Dec 12 02:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 slot1_33 slot1_33         25 Dec 12 02:56 all-input-dids.txt
-rw-r--r-- 1 slot1_33 slot1_33        519 Dec 12 02:56 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_33 slot1_33       3574 Dec 12 02:56 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_33 slot1_33     204800 Dec 12 02:56 mem.db
-rw-r--r-- 1 slot1_33 slot1_33      36864 Dec 12 02:56 time.db
-rw-r--r-- 1 slot1_33 slot1_33 6323798461 Dec 12 02:56 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_261437_106_1_20251212T073610Z_detsim_reco1.root
-rw-r--r-- 1 slot1_33 slot1_33   49289700 Dec 12 02:55 Pandora_Events.pndr
-rw-r--r-- 1 slot1_33 slot1_33          0 Dec 12 02:35 ProtoDUNE.log
-rw-r--r-- 1 slot1_33 slot1_33          0 Dec 12 02:32 debugprod.log
-rw-r--r-- 1 slot1_33 slot1_33        519 Dec 12 02:32 detsim_single_protodunehd_hist.root
justIN time: 2025-12-19 07:35:51 UTC       justIN version: 01.05.03