justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.32@dunegpschedd02.fnal.gov

Jobsub ID234970.32@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:55:33
From worker nodeHostnamedunegli-7568269-0-fnpc23016.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:51
Input filesusertests:pdhd_prod_beam__226242_29_1_20251008T182708Z_gen_g4_IonScintPDExt.root_227038_6_1_20251010T161216Z_PDInt.root
JobscriptExit code0
Real time1h (3792s)
CPU time1h (3720s = 98%)
Max RSS bytes7097790464 (6768 MiB)
Outputting started2025-10-29 20:41:03
Output files
Finished2025-10-29 20:55:33
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

XVs -> 681 XUVs
2134 XUVs total
914 collection wire objects
2134 potential space points
Neighbour search...
45718 tests to find 28620 neighbours
Iterating with no regularization...
Begin: 1.18518e+07
0 1.13319e+07
1 1.12472e+07
2 1.12369e+07
Now with regularization...
Begin: 1.04699e+07
0 1.04673e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:38:04 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15619!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 94889 traces, 94889 summary
Retagger: tagging trace set: gauss with 75182 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75182 traces tagged "gauss"
FrameSaver: q=2.30257e+06 n=1171623 tag=gauss
wclsFrameSaver: saving 94889 traces tagged "wiener"
FrameSaver: q=2.92194e+06 n=1138114 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12055 XUs and 9882 XVs -> 93 XUVs
C:0 T:1 52691 XUs and 47143 XVs -> 1540 XUVs
C:0 T:2 35530 XUs and 54776 XVs -> 1908 XUVs
C:0 T:3 14 XUs and 9 XVs -> 0 XUVs
C:0 T:4 6 XUs and 15 XVs -> 2 XUVs
C:0 T:5 27988 XUs and 52202 XVs -> 4518 XUVs
C:0 T:6 26747 XUs and 44321 XVs -> 462 XUVs
C:0 T:7 5 XUs and 9 XVs -> 0 XUVs
8523 XUVs total
3024 collection wire objects
8523 potential space points
Neighbour search...
522843 tests to find 280764 neighbours
Iterating with no regularization...
Begin: 4.49728e+07
0 3.88341e+07
1 3.84112e+07
2 3.83348e+07
3 3.83097e+07
Now with regularization...
Begin: 3.61444e+07
0 3.61267e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:39:30 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14189!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 96357 traces, 96357 summary
Retagger: tagging trace set: gauss with 75730 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75730 traces tagged "gauss"
FrameSaver: q=2.4401e+06 n=1179674 tag=gauss
wclsFrameSaver: saving 96357 traces tagged "wiener"
FrameSaver: q=3.18034e+06 n=1160457 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9571 XUs and 11087 XVs -> 131 XUVs
C:0 T:1 54018 XUs and 79772 XVs -> 3439 XUVs
C:0 T:2 49475 XUs and 44720 XVs -> 1268 XUVs
C:0 T:3 24 XUs and 5 XVs -> 0 XUVs
C:0 T:4 87 XUs and 91 XVs -> 13 XUVs
C:0 T:5 27291 XUs and 31394 XVs -> 1746 XUVs
C:0 T:6 81480 XUs and 66497 XVs -> 3487 XUVs
10084 XUVs total
3170 collection wire objects
10084 potential space points
Neighbour search...
804132 tests to find 403110 neighbours
Iterating with no regularization...
Begin: 7.36896e+07
0 6.51416e+07
1 6.46661e+07
2 6.46119e+07
Now with regularization...
Begin: 6.03805e+07
0 6.03514e+07
29-Oct-2025 20:41:03 UTC  Closed output file "pdhd_prod_beam__226242_29_1_20251008T182708Z_gen_g4_IonScintPDExt.root_227038_6_1_20251010T161216Z_PDInt.root_234970_32_1_20251029T193753Z_detsim_reco1.root"
29-Oct-2025 20:41:03 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            54.9573       91.8815       150.614       87.408        24.0665        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000416776   0.00388671    0.00564582     0.0041524    0.00157018       10     
reco:opdec:Deconvolution                             0.427001      0.592558      0.664238      0.609759      0.0642577       10     
reco:ophitspe:OpHitFinderDeco                        0.0976702     0.103912      0.108153      0.103873     0.00309653       10     
reco:opflash:OpFlashFinder                           0.0432496     0.0719875     0.0950186     0.0777281     0.0147149       10     
reco:opslicer:OpSlicer                               0.104823       0.17397      0.218779      0.174984      0.0301377       10     
reco:wclsdatahd:WireCellToolkit                       31.6081       32.2878       32.7278       32.3048       0.30386        10     
reco:gaushit:GausHitFinder                           0.277963      0.522666      0.687001      0.530371      0.103771        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000125152   0.000235178   0.000437296   0.000225175   7.5523e-05       10     
reco:reco3d:SpacePointSolver                         0.132515      0.594333       1.01395      0.597549      0.269264        10     
reco:hitpdune:DisambigFromSpacePoints                0.105738       0.45159      0.701187       0.47158      0.166011        10     
reco:pandora:StandardPandora                          3.6272        19.516        67.5314       14.6182       17.0727        10     
reco:pandoraWriter:StandardPandora                   0.0835924     0.166007      0.238947      0.162485      0.0407189       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.534937       1.14653       1.57798       1.23451      0.311105        10     
reco:pandoraShower:LArPandoraShowerCreation          0.408461       1.11271       1.53352       1.15545      0.347959        10     
reco:pandoracalo:Calorimetry                          0.26591      0.628438      0.891546      0.665632      0.175202        10     
reco:pandoracalonosce:Calorimetry                    0.262677      0.616199      0.884405      0.652191      0.172418        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.390235       1.14199       1.63648       1.18219      0.367048        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.380624       1.08729       1.5761        1.1262       0.346595        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0215477     0.0337393     0.0388041     0.0351503     0.0050347       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0205235     0.0321376     0.0377795     0.0340068    0.00466676       10     
reco:emtrkmichelid:EmTrackMichelId                    11.715        21.1056       26.9055       21.8781       3.87146        10     
reco:crttag:SingleCRTMatchingProducer               0.00266809    0.00381706    0.00547019    0.00353222    0.000848109      10     
reco:crtreco:TwoCRTMatchingProducer                  0.0021646    0.00280272    0.00318594    0.00282572    0.000284638      10     
reco:pandora2Track:LArPandoraTrackCreation            1.06533       2.80873       3.88715       2.92393      0.868706        10     
reco:pandora2calo:ShowerCalorimetry                  0.341904       1.0118        1.43509       1.04329      0.323512        10     
reco:pandora2calonosce:ShowerCalorimetry             0.322226      0.995392       1.40565       1.03406      0.323748        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.020833      0.0324299     0.0371501     0.0329324    0.00479462       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0200671     0.0315303     0.0385067     0.0313802    0.00491358       10     
reco:pandora2Shower:LArPandoraShowerCreation          0.40925       1.26166       1.70499       1.33418      0.396225        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.442849       1.39367       1.99089       1.46128      0.450003        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        0.42467       1.36222       1.94168       1.42916      0.440159        10     
[art]:TriggerResults:TriggerResultInserter          1.7544e-05    2.97075e-05   0.000102919   2.16065e-05   2.4629e-05       10     
end_path:out1:RootOutput                             3.808e-06    8.2902e-06    3.0058e-05     4.534e-06    7.6159e-06       10     
end_path:out1:RootOutput(write)                      0.970123       1.52125       1.8862        1.48534      0.246377        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6379.36 MB
  Peak resident set size usage (VmHWM): 4170.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11846500
-rw-r--r-- 1 dunegli fnalgrid      38708 Oct 29 20:41 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        199 Oct 29 20:41 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:41 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:41 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:41 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:41 time.db
-rw-r--r-- 1 dunegli fnalgrid 6214665507 Oct 29 20:41 pdhd_prod_beam__226242_29_1_20251008T182708Z_gen_g4_IonScintPDExt.root_227038_6_1_20251010T161216Z_PDInt.root_234970_32_1_20251029T193753Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   53895620 Oct 29 20:40 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:27 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:24 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:24 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5861939970 Oct 29 20:24 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-03 18:32:38 UTC       justIN version: 01.05.01