justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 240536.0@dunegpschedd01.fnal.gov

Jobsub ID240536.0@dunegpschedd01.fnal.gov
Workflow ID9383
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 19:50:48
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2025-10-29 23:22:08
From worker nodeHostnamemwt2-c067.campuscluster.illinois.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-29 21:24:44
Input filesusertests:pdhd_prod_beam__226698_42_1_20251009T171316Z_gen_g4_IonScintPDExt.root_226750_122_1_20251009T211813Z_PDInt.root
JobscriptExit code0
Real time1h (6831s)
CPU time1h (6631s = 97%)
Max RSS bytes8431284224 (8040 MiB)
Outputting started2025-10-29 23:18:36
Output filesusertests:pdhd_prod_beam__226698_42_1_20251009T171316Z_gen_g4_IonScintPDExt.root_226750_122_1_20251009T211813Z_PDInt.root_240536_0_1_20251029T212449Z_detsim_reco1.root
Finished2025-10-29 23:22:08
Saved logsjustin-logs:240536.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Us and 30928 XVs -> 1205 XUVs
5948 XUVs total
2612 collection wire objects
5948 potential space points
Neighbour search...
359774 tests to find 111864 neighbours
Iterating with no regularization...
Begin: 5.75229e+07
0 5.24597e+07
1 5.20247e+07
2 5.19895e+07
Now with regularization...
Begin: 5.00042e+07
0 4.99958e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 78 at 29-Oct-2025 18:12:49 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12615!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 84528 traces, 84528 summary
Retagger: tagging trace set: gauss with 69736 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 69736 traces tagged "gauss"
FrameSaver: q=2.09347e+06 n=1036078 tag=gauss
wclsFrameSaver: saving 84528 traces tagged "wiener"
FrameSaver: q=2.75647e+06 n=1025835 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11624 XUs and 9467 XVs -> 92 XUVs
C:0 T:1 78440 XUs and 65498 XVs -> 1774 XUVs
C:0 T:2 46912 XUs and 29131 XVs -> 2160 XUVs
C:0 T:3 19 XUs and 11 XVs -> 0 XUVs
C:0 T:4 4 XUs and 7 XVs -> 0 XUVs
C:0 T:5 21888 XUs and 9293 XVs -> 206 XUVs
C:0 T:6 40151 XUs and 19992 XVs -> 853 XUVs
C:0 T:7 12 XUs and 9 XVs -> 0 XUVs
5085 XUVs total
2310 collection wire objects
5085 potential space points
Neighbour search...
203485 tests to find 112556 neighbours
Iterating with no regularization...
Begin: 4.4371e+07
0 4.10362e+07
1 4.08669e+07
2 4.085e+07
Now with regularization...
Begin: 3.91421e+07
0 3.91353e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 79 at 29-Oct-2025 18:15:08 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 19292!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 104994 traces, 104994 summary
Retagger: tagging trace set: gauss with 85670 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 85670 traces tagged "gauss"
FrameSaver: q=2.892e+06 n=1423148 tag=gauss
wclsFrameSaver: saving 104994 traces tagged "wiener"
FrameSaver: q=3.72946e+06 n=1380947 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 17308 XUs and 12067 XVs -> 154 XUVs
C:0 T:1 105713 XUs and 78126 XVs -> 3413 XUVs
C:0 T:2 25419 XUs and 16615 XVs -> 477 XUVs
C:0 T:4 25 XUs and 13 XVs -> 0 XUVs
C:0 T:5 30176 XUs and 18838 XVs -> 723 XUVs
C:0 T:6 36174 XUs and 32641 XVs -> 1426 XUVs
C:0 T:7 10 XUs and 18 XVs -> 4 XUVs
6197 XUVs total
3140 collection wire objects
6197 potential space points
Neighbour search...
181179 tests to find 72786 neighbours
Iterating with no regularization...
Begin: 5.53227e+07
0 5.16033e+07
1 5.12911e+07
2 5.12716e+07
Now with regularization...
Begin: 4.91472e+07
0 4.91407e+07
29-Oct-2025 18:18:34 CDT  Closed output file "pdhd_prod_beam__226698_42_1_20251009T171316Z_gen_g4_IonScintPDExt.root_226750_122_1_20251009T211813Z_PDInt.root_240536_0_1_20251029T212449Z_detsim_reco1.root"
29-Oct-2025 18:18:34 CDT  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            123.343       184.893       251.159       177.801       39.5407        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00125268     0.0158365     0.0677072     0.0111117     0.0176684       10     
reco:opdec:Deconvolution                             0.756291       1.00108       1.17326       1.02377      0.120838        10     
reco:ophitspe:OpHitFinderDeco                        0.224262      0.245911      0.281275      0.241627      0.0187522       10     
reco:opflash:OpFlashFinder                           0.0602067     0.102594      0.136058      0.114789      0.0265066       10     
reco:opslicer:OpSlicer                               0.163736      0.317539       0.43806       0.32283      0.0745054       10     
reco:wclsdatahd:WireCellToolkit                       47.8906       50.5056       54.7881       49.652        2.29281        10     
reco:gaushit:GausHitFinder                           0.886682       1.17651       1.45931       1.25284      0.193745        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000386655   0.00049178    0.000726543   0.000459572   0.000114553      10     
reco:reco3d:SpacePointSolver                          1.05636       1.53608       3.08596       1.32387      0.560876        10     
reco:hitpdune:DisambigFromSpacePoints                0.529032       1.20654       2.20106       1.22493       0.46967        10     
reco:pandora:StandardPandora                          17.2602       38.1787       55.5934       37.2155       11.876         10     
reco:pandoraWriter:StandardPandora                   0.364162      0.462279      0.601306      0.447149      0.075553        10     
reco:pandoraTrack:LArPandoraTrackCreation             2.2837        4.2885        7.22601       3.60035       1.70755        10     
reco:pandoraShower:LArPandoraShowerCreation           1.81514       4.26133       7.91491       3.66523       1.94762        10     
reco:pandoracalo:Calorimetry                          1.06785       2.19455       3.90665       1.85249      0.999767        10     
reco:pandoracalonosce:Calorimetry                      1.205        2.2112        3.9142        1.71394       1.00001        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.75433       4.08759       8.40603       3.67492       2.07617        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.48481       3.62894       7.40673       3.35907       1.79112        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0560922     0.0803072     0.106713      0.0867181     0.0176028       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0542536     0.078595      0.106346      0.0817835     0.0180895       10     
reco:emtrkmichelid:EmTrackMichelId                    28.343        37.9114       45.1303       40.3931       5.81593        10     
reco:crttag:SingleCRTMatchingProducer               0.00667454     0.0113819     0.0176885     0.0118282    0.00304176       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00529646    0.00956369     0.0178674     0.0101034     0.003502        10     
reco:pandora2Track:LArPandoraTrackCreation            4.91641       9.83961       16.8196       8.68442       3.87239        10     
reco:pandora2calo:ShowerCalorimetry                   1.2745        3.27922       6.2352        2.94911       1.58775        10     
reco:pandora2calonosce:ShowerCalorimetry              1.42079       3.04066       5.88767       2.76675       1.33625        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0533546     0.0767528     0.102024      0.0807336     0.0154308       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0529531     0.0752513     0.0972163     0.0786224     0.0148431       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.69014       3.83182       6.74617       3.93138       1.53283        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.86097       4.09161       7.81988       3.8131        1.81021        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.68769       3.9792        7.8956        3.58719       1.9131         10     
[art]:TriggerResults:TriggerResultInserter          4.2379e-05    7.09931e-05   0.000152857   6.48065e-05   3.21231e-05      10     
end_path:out1:RootOutput                            1.0479e-05    1.62667e-05    3.753e-05    1.4558e-05    7.2546e-06       10     
end_path:out1:RootOutput(write)                       2.24877       2.94515       3.5156        3.02129      0.410846        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6558.93 MB
  Peak resident set size usage (VmHWM): 4328.95 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 13649072
-rw-r--r-- 1 dune osgvo      39486 Oct 29 18:18 jobscript.log
-rw-r--r-- 1 dune osgvo        201 Oct 29 18:18 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Oct 29 18:18 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Oct 29 18:18 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     200704 Oct 29 18:18 mem.db
-rw-r--r-- 1 dune osgvo      36864 Oct 29 18:18 time.db
-rw-r--r-- 1 dune osgvo 7146135013 Oct 29 18:18 pdhd_prod_beam__226698_42_1_20251009T171316Z_gen_g4_IonScintPDExt.root_226750_122_1_20251009T211813Z_PDInt.root_240536_0_1_20251029T212449Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   57062740 Oct 29 18:16 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Oct 29 17:48 ProtoDUNE.log
-rw-r--r-- 1 dune osgvo          0 Oct 29 17:45 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Oct 29 17:45 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 6773129114 Oct 29 17:45 temp_detsim.root
-rw-r--r-- 1 dune osgvo        122 Oct 29 16:24 all-input-dids.txt
justIN time: 2025-11-05 21:12:56 UTC       justIN version: 01.05.01