justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.27@dunegpschedd02.fnal.gov

Jobsub ID234970.27@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:58:55
From worker nodeHostnamedunegli-7568310-0-fnpc19102.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:38:18
Input filesusertests:pdhd_prod_beam__231691_51_1_20251008T183154Z_gen_g4_IonScintPDExt.root_227038_121_1_20251010T163752Z_PDInt.root
JobscriptExit code0
Real time1h (4434s)
CPU time1h (4347s = 98%)
Max RSS bytes6328606720 (6035 MiB)
Outputting started2025-10-29 20:52:12
Output files
Finished2025-10-29 20:58:55
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

hbours
Iterating with no regularization...
Begin: 1.9502e+07
0 1.70977e+07
1 1.68975e+07
2 1.68556e+07
3 1.68435e+07
Now with regularization...
Begin: 1.54061e+07
0 1.53972e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:48:47 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12083!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 72362 traces, 72362 summary
Retagger: tagging trace set: gauss with 58314 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 58314 traces tagged "gauss"
FrameSaver: q=1.52182e+06 n=839288 tag=gauss
wclsFrameSaver: saving 72362 traces tagged "wiener"
FrameSaver: q=1.95393e+06 n=819273 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 6947 XUs and 4452 XVs -> 68 XUVs
C:0 T:1 64908 XUs and 36163 XVs -> 2397 XUVs
C:0 T:2 11476 XUs and 18873 XVs -> 623 XUVs
C:0 T:5 11717 XUs and 8162 XVs -> 540 XUVs
C:0 T:6 15926 XUs and 24706 XVs -> 370 XUVs
3998 XUVs total
1751 collection wire objects
3998 potential space points
Neighbour search...
267148 tests to find 85804 neighbours
Iterating with no regularization...
Begin: 4.86793e+07
0 4.40283e+07
1 4.36146e+07
2 4.35927e+07
Now with regularization...
Begin: 4.19467e+07
0 4.19399e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:50:06 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14018!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 106175 traces, 106175 summary
Retagger: tagging trace set: gauss with 83147 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 83147 traces tagged "gauss"
FrameSaver: q=2.47941e+06 n=1301381 tag=gauss
wclsFrameSaver: saving 106175 traces tagged "wiener"
FrameSaver: q=3.06852e+06 n=1240114 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14542 XUs and 13079 XVs -> 143 XUVs
C:0 T:1 85635 XUs and 72123 XVs -> 1545 XUVs
C:0 T:2 44951 XUs and 29479 XVs -> 1117 XUVs
C:0 T:4 18 XUs and 19 XVs -> 0 XUVs
C:0 T:5 60863 XUs and 65149 XVs -> 1006 XUVs
C:0 T:6 27109 XUs and 13500 XVs -> 686 XUVs
4497 XUVs total
2507 collection wire objects
4497 potential space points
Neighbour search...
147733 tests to find 83832 neighbours
Iterating with no regularization...
Begin: 2.61327e+07
0 2.43913e+07
1 2.42949e+07
2 2.42834e+07
Now with regularization...
Begin: 2.32667e+07
0 2.32638e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:52:11 UTC  Closed output file "pdhd_prod_beam__231691_51_1_20251008T183154Z_gen_g4_IonScintPDExt.root_227038_121_1_20251010T163752Z_PDInt.root_234970_27_1_20251029T193820Z_detsim_reco1.root"
29-Oct-2025 20:52:11 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            68.4904       100.926       147.849       96.7848       25.4639        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000512514   0.00403565    0.00608889    0.00499553    0.00183399       10     
reco:opdec:Deconvolution                             0.551902      0.707021      0.832496      0.725861      0.0904976       10     
reco:ophitspe:OpHitFinderDeco                        0.119422       0.12624      0.134816      0.125307     0.00565313       10     
reco:opflash:OpFlashFinder                            0.0442       0.0647458     0.0810926     0.0642161     0.0132301       10     
reco:opslicer:OpSlicer                               0.124886      0.189101      0.263853      0.186602      0.0395584       10     
reco:wclsdatahd:WireCellToolkit                       36.8105       39.9395       45.1807       39.4817       2.3893         10     
reco:gaushit:GausHitFinder                           0.372337      0.562791      0.874642      0.531794      0.160714        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000189811   0.000330123   0.000590445   0.000307738   0.000122916      10     
reco:reco3d:SpacePointSolver                         0.352453      0.837688       1.75014      0.695812      0.490469        10     
reco:hitpdune:DisambigFromSpacePoints                0.188921      0.489408       1.0347       0.486544      0.259865        10     
reco:pandora:StandardPandora                          4.76826       15.8088       37.6617       12.4264       9.73743        10     
reco:pandoraWriter:StandardPandora                   0.124642      0.202935      0.313582      0.196194      0.0635585       10     
reco:pandoraTrack:LArPandoraTrackCreation             0.4827        1.23601       2.28949       1.13448      0.600415        10     
reco:pandoraShower:LArPandoraShowerCreation           0.49365       1.29826       2.54353       1.11283      0.727336        10     
reco:pandoracalo:Calorimetry                         0.272696      0.606273       1.03898      0.566973      0.260682        10     
reco:pandoracalonosce:Calorimetry                    0.267979      0.605053       1.11739      0.561913      0.271181        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.472246       1.27704       2.95416       1.06658      0.802681        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.454718       1.1735        2.44891      0.995512      0.685845        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0321152     0.0441604     0.0586453      0.04169      0.0098963       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.031932      0.0433618     0.0584846     0.0411729    0.00968767       10     
reco:emtrkmichelid:EmTrackMichelId                    16.2536       23.6358       30.5851       21.7201       5.31452        10     
reco:crttag:SingleCRTMatchingProducer               0.00260436    0.00430271    0.00666732    0.00428459    0.00124165       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00225818    0.00370352    0.00472687    0.00403764    0.000958771      10     
reco:pandora2Track:LArPandoraTrackCreation            1.34828       3.39258       6.29703       2.91463       1.8585         10     
reco:pandora2calo:ShowerCalorimetry                  0.435307       1.09928       2.34787      0.917326      0.647772        10     
reco:pandora2calonosce:ShowerCalorimetry             0.428815       1.11419        2.563       0.899915      0.715034        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0316037     0.044397      0.0615473     0.0429885     0.0104329       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0310899     0.0435181     0.0615528     0.0405332     0.0104912       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.605995       1.56534       3.28505       1.29587       0.94257        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.594871       1.47333       3.01996       1.25769      0.849764        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.587319       1.42806       2.78351       1.24679      0.801957        10     
[art]:TriggerResults:TriggerResultInserter           1.617e-05    2.51682e-05   8.1711e-05     1.954e-05    1.89356e-05      10     
end_path:out1:RootOutput                             4.07e-06     8.5321e-06    2.5001e-05     8.285e-06    5.91547e-06      10     
end_path:out1:RootOutput(write)                       1.32685       1.82127       2.45189       1.93361      0.387156        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6115.94 MB
  Peak resident set size usage (VmHWM): 3907.85 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11029264
-rw-r--r-- 1 dunegli fnalgrid      42497 Oct 29 20:52 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:52 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:52 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:52 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:52 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:52 time.db
-rw-r--r-- 1 dunegli fnalgrid 5784228537 Oct 29 20:52 pdhd_prod_beam__231691_51_1_20251008T183154Z_gen_g4_IonScintPDExt.root_227038_121_1_20251010T163752Z_PDInt.root_234970_27_1_20251029T193820Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   49162870 Oct 29 20:51 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:36 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:33 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:33 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5460266979 Oct 29 20:33 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:38 all-input-dids.txt
justIN time: 2025-11-03 18:32:41 UTC       justIN version: 01.05.01