justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242475.60@dunegpschedd01.fnal.gov

Jobsub ID242475.60@dunegpschedd01.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 18:03:07
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-03 20:36:13
From worker nodeHostnamee2597
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 18:18:57
Input filesusertests:pdhd_prod_beam__226470_0_1_20251008T212416Z_gen_g4_IonScintPDExt.root_227040_70_1_20251010T170129Z_PDInt.root
JobscriptExit code0
Real time2h (8033s)
CPU time2h (7396s = 92%)
Max RSS bytes7640936448 (7286 MiB)
Outputting started2025-11-03 20:32:51
Output filesusertests:pdhd_prod_beam__226470_0_1_20251008T212416Z_gen_g4_IonScintPDExt.root_227040_70_1_20251010T170129Z_PDInt.root_242475_60_1_20251103T181907Z_detsim_reco1.root
Finished2025-11-03 20:36:13
Saved logsjustin-logs:242475.60-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2666 collection wire objects
7026 potential space points
Neighbour search...
366932 tests to find 221288 neighbours
Iterating with no regularization...
Begin: 3.74931e+07
0 3.40215e+07
1 3.37432e+07
2 3.37033e+07
3 3.36893e+07
Now with regularization...
Begin: 3.15201e+07
0 3.15105e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 03-Nov-2025 14:29:05 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 18192!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 104641 traces, 104641 summary
Retagger: tagging trace set: gauss with 82069 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 82069 traces tagged "gauss"
FrameSaver: q=2.55595e+06 n=1340519 tag=gauss
wclsFrameSaver: saving 104641 traces tagged "wiener"
FrameSaver: q=3.26598e+06 n=1293493 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11477 XUs and 10772 XVs -> 114 XUVs
C:0 T:1 66141 XUs and 66817 XVs -> 3649 XUVs
C:0 T:2 35825 XUs and 49159 XVs -> 1282 XUVs
C:0 T:4 8 XUs and 15 XVs -> 0 XUVs
C:0 T:5 51104 XUs and 54894 XVs -> 1669 XUVs
C:0 T:6 27158 XUs and 65497 XVs -> 1511 XUVs
C:0 T:7 34 XUs and 52 XVs -> 0 XUVs
8225 XUVs total
2958 collection wire objects
8225 potential space points
Neighbour search...
360363 tests to find 191810 neighbours
Iterating with no regularization...
Begin: 4.13277e+07
0 3.80854e+07
1 3.78223e+07
2 3.77834e+07
3 3.77735e+07
Now with regularization...
Begin: 3.57719e+07
0 3.57661e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 14:31:25 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12893!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 67508 traces, 67508 summary
Retagger: tagging trace set: gauss with 57207 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 57207 traces tagged "gauss"
FrameSaver: q=1.41427e+06 n=855829 tag=gauss
wclsFrameSaver: saving 67508 traces tagged "wiener"
FrameSaver: q=1.78403e+06 n=842596 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 5797 XUs and 3614 XVs -> 19 XUVs
C:0 T:1 19310 XUs and 11731 XVs -> 355 XUVs
C:0 T:2 10077 XUs and 15135 XVs -> 677 XUVs
C:0 T:4 9 XUs and 18 XVs -> 0 XUVs
C:0 T:5 19711 XUs and 24020 XVs -> 842 XUVs
C:0 T:6 10832 XUs and 11537 XVs -> 389 XUVs
2282 XUVs total
1358 collection wire objects
2282 potential space points
Neighbour search...
43412 tests to find 25300 neighbours
Iterating with no regularization...
Begin: 1.48569e+07
0 1.3822e+07
1 1.37826e+07
2 1.37788e+07
Now with regularization...
Begin: 1.29271e+07
0 1.29259e+07
03-Nov-2025 14:32:50 CST  Closed output file "pdhd_prod_beam__226470_0_1_20251008T212416Z_gen_g4_IonScintPDExt.root_227040_70_1_20251010T170129Z_PDInt.root_242475_60_1_20251103T181907Z_detsim_reco1.root"
03-Nov-2025 14:32:50 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            81.6324       258.608       893.727       138.008       244.547        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000676736    0.0153701     0.0805564    0.00903391     0.022206        10     
reco:opdec:Deconvolution                             0.550319      0.941293       1.97841      0.842141      0.381514        10     
reco:ophitspe:OpHitFinderDeco                        0.122656      0.190659      0.396732      0.138494      0.0929426       10     
reco:opflash:OpFlashFinder                           0.0510084     0.107304      0.244886      0.0950399     0.0504839       10     
reco:opslicer:OpSlicer                               0.136996      0.271373      0.500645      0.242224       0.10114        10     
reco:wclsdatahd:WireCellToolkit                       40.5728       62.6223       149.551       43.5483       39.3823        10     
reco:gaushit:GausHitFinder                            0.46882       1.12213       4.0746       0.795189       1.0161         10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000188035   0.00189032    0.00964568    0.000378769    0.0031766       10     
reco:reco3d:SpacePointSolver                         0.208531       1.94278       11.0972      0.879517       3.10101        10     
reco:hitpdune:DisambigFromSpacePoints                0.158054       1.35568       7.37097      0.695816       2.04112        10     
reco:pandora:StandardPandora                          6.82755       63.2935       312.382       23.8486       90.5497        10     
reco:pandoraWriter:StandardPandora                   0.138155      0.793258       2.91642      0.295526      0.905134        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.69703       8.81226       39.361        1.82112       13.0291        10     
reco:pandoraShower:LArPandoraShowerCreation          0.576641       4.37793       19.5491       2.32602       5.39228        10     
reco:pandoracalo:Calorimetry                         0.390981       2.29182       11.6951       1.13022       3.21618        10     
reco:pandoracalonosce:Calorimetry                     0.37833       2.46869       14.0945       1.13488       3.92614        10     
reco:pandoraShowercalo:ShowerCalorimetry              0.45202       4.21378       21.8198       2.52269       5.97511        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.429375       5.24128       18.1049       2.29769       6.50427        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0269218     0.113241      0.501163      0.0539422     0.137927        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.027198      0.111993       0.4976       0.0537265     0.135473        10     
reco:emtrkmichelid:EmTrackMichelId                    20.5281       61.5954       156.313       36.005        47.3204        10     
reco:crttag:SingleCRTMatchingProducer                0.0033155     0.0153064     0.0452949    0.00668582     0.015459        10     
reco:crtreco:TwoCRTMatchingProducer                 0.00276311     0.0108098     0.0339666    0.00494749     0.0109981       10     
reco:pandora2Track:LArPandoraTrackCreation            1.31966       17.8628       90.2738       5.68048       26.5052        10     
reco:pandora2calo:ShowerCalorimetry                  0.356533       2.57807       6.86077       2.04588       1.88532        10     
reco:pandora2calonosce:ShowerCalorimetry             0.350561       2.41983       6.23022       1.98787       1.69945        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0245166     0.0719711     0.147872      0.056201      0.0410311       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0239274     0.0671886     0.124599      0.0553639     0.0351039       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.604237       3.93891       11.4841       2.48709       3.49823        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.569416       3.18798       8.51885       2.71285       2.26681        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.540957       3.21655       7.17107       2.57382       2.1441         10     
[art]:TriggerResults:TriggerResultInserter          2.2994e-05    5.9309e-05    0.000179979   3.42095e-05   4.96111e-05      10     
end_path:out1:RootOutput                             7.064e-06    1.76643e-05    5.326e-05    1.13365e-05   1.37307e-05      10     
end_path:out1:RootOutput(write)                       1.45179       3.08377       5.96444       2.37574       1.5049         10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6342.51 MB
  Peak resident set size usage (VmHWM): 4086.92 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12612684
-rw-r--r-- 1 slot1_42 slot1_42      39172 Nov  3 14:32 jobscript.log
-rw-r--r-- 1 slot1_42 slot1_42        199 Nov  3 14:32 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_42 slot1_42        519 Nov  3 14:32 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_42 slot1_42       3574 Nov  3 14:32 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_42 slot1_42     200704 Nov  3 14:32 mem.db
-rw-r--r-- 1 slot1_42 slot1_42      36864 Nov  3 14:32 time.db
-rw-r--r-- 1 slot1_42 slot1_42 6619015022 Nov  3 14:32 pdhd_prod_beam__226470_0_1_20251008T212416Z_gen_g4_IonScintPDExt.root_227040_70_1_20251010T170129Z_PDInt.root_242475_60_1_20251103T181907Z_detsim_reco1.root
-rw-r--r-- 1 slot1_42 slot1_42   58637720 Nov  3 14:32 Pandora_Events.pndr
-rw-r--r-- 1 slot1_42 slot1_42          0 Nov  3 13:50 ProtoDUNE.log
-rw-r--r-- 1 slot1_42 slot1_42          0 Nov  3 13:47 debugprod.log
-rw-r--r-- 1 slot1_42 slot1_42        519 Nov  3 13:47 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 slot1_42 slot1_42 6237404922 Nov  3 13:47 temp_detsim.root
-rw-r--r-- 1 slot1_42 slot1_42        120 Nov  3 12:19 all-input-dids.txt
justIN time: 2025-11-04 01:54:56 UTC       justIN version: 01.05.01