justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236804.29@dunegpschedd02.fnal.gov

Jobsub ID236804.29@dunegpschedd02.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 18:19:09
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-03 20:25:40
From worker nodeHostnamee2600
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 18:53:29
Input filesusertests:pdhd_prod_beam__231896_144_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_156_1_20251010T171938Z_PDInt.root
JobscriptExit code0
Real time1h (5315s)
CPU time1h (4915s = 92%)
Max RSS bytes6582149120 (6277 MiB)
Outputting started2025-11-03 20:22:11
Output filesusertests:pdhd_prod_beam__231896_144_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_156_1_20251010T171938Z_PDInt.root_236804_29_1_20251103T185346Z_detsim_reco1.root
Finished2025-11-03 20:25:40
Saved logsjustin-logs:236804.29-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2.84313e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 03-Nov-2025 14:15:53 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14208!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 92780 traces, 92780 summary
Retagger: tagging trace set: gauss with 76499 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76499 traces tagged "gauss"
FrameSaver: q=2.486e+06 n=1285325 tag=gauss
wclsFrameSaver: saving 92780 traces tagged "wiener"
FrameSaver: q=3.10661e+06 n=1240442 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11065 XUs and 7922 XVs -> 120 XUVs
C:0 T:1 81485 XUs and 65338 XVs -> 3998 XUVs
C:0 T:2 24656 XUs and 27691 XVs -> 1711 XUVs
C:0 T:3 15 XUs and 55 XVs -> 8 XUVs
C:0 T:5 45307 XUs and 23541 XVs -> 1020 XUVs
C:0 T:6 19742 XUs and 18090 XVs -> 781 XUVs
7638 XUVs total
3068 collection wire objects
7638 potential space points
Neighbour search...
498668 tests to find 177810 neighbours
Iterating with no regularization...
Begin: 5.20588e+07
0 4.66972e+07
1 4.62167e+07
2 4.61686e+07
3 4.6158e+07
Now with regularization...
Begin: 4.41272e+07
0 4.41194e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 14:19:15 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12798!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 93456 traces, 93456 summary
Retagger: tagging trace set: gauss with 74139 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74139 traces tagged "gauss"
FrameSaver: q=2.0859e+06 n=1128677 tag=gauss
wclsFrameSaver: saving 93456 traces tagged "wiener"
FrameSaver: q=2.73358e+06 n=1099430 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9379 XUs and 8796 XVs -> 43 XUVs
C:0 T:1 42592 XUs and 39952 XVs -> 1531 XUVs
C:0 T:2 13651 XUs and 81614 XVs -> 249 XUVs
C:0 T:3 10 XUs and 32 XVs -> 1 XUVs
C:0 T:5 60577 XUs and 69597 XVs -> 2713 XUVs
C:0 T:6 48676 XUs and 67074 XVs -> 1197 XUVs
5734 XUVs total
2341 collection wire objects
5734 potential space points
Neighbour search...
240384 tests to find 108700 neighbours
Iterating with no regularization...
Begin: 3.97613e+07
0 3.5585e+07
1 3.52726e+07
2 3.52332e+07
3 3.52201e+07
Now with regularization...
Begin: 3.39348e+07
0 3.39267e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 14:21:57 CST  Closed output file "pdhd_prod_beam__231896_144_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_156_1_20251010T171938Z_PDInt.root_236804_29_1_20251103T185346Z_detsim_reco1.root"
03-Nov-2025 14:21:57 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            124.232       162.641       195.469       162.839       19.6903        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000800883   0.00877175     0.0159962    0.00898945    0.00511957       10     
reco:opdec:Deconvolution                              0.76253      0.978192       1.36383      0.948581      0.168356        10     
reco:ophitspe:OpHitFinderDeco                        0.148423      0.228425      0.302865      0.249674      0.0535661       10     
reco:opflash:OpFlashFinder                           0.060694      0.0993419     0.150058      0.0954595     0.0262826       10     
reco:opslicer:OpSlicer                               0.221954      0.301465       0.46863      0.278574      0.0722515       10     
reco:wclsdatahd:WireCellToolkit                       49.3807       56.3471       65.0534       56.6519       4.66796        10     
reco:gaushit:GausHitFinder                           0.669883      0.993609       1.35153      0.928239      0.223802        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000287392   0.00041188    0.000632024   0.000416217   9.36571e-05      10     
reco:reco3d:SpacePointSolver                         0.477109       0.97987       1.33774      0.961983       0.27522        10     
reco:hitpdune:DisambigFromSpacePoints                0.459297      0.768321       1.2476       0.733009      0.262831        10     
reco:pandora:StandardPandora                          13.7355       25.8094       39.5927       24.4275       7.07317        10     
reco:pandoraWriter:StandardPandora                   0.303574      0.447315      0.608084      0.415393      0.102228        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.37428       2.10173       3.03996       1.99883      0.509109        10     
reco:pandoraShower:LArPandoraShowerCreation          0.990228       1.91597       3.42717       1.81436      0.638463        10     
reco:pandoracalo:Calorimetry                         0.707251       1.08451       1.4997        1.04934      0.251845        10     
reco:pandoracalonosce:Calorimetry                    0.711589       1.05837       1.45052       1.03809      0.232276        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.00171       1.90881       3.14601       1.87908      0.575824        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.957902       1.80025       2.97785       1.74855      0.540239        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0469394     0.0661967     0.0926934     0.0637634     0.0154427       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0472304     0.0687677     0.100312      0.0690739     0.0163938       10     
reco:emtrkmichelid:EmTrackMichelId                    31.7298       44.798        52.7256       46.2878       6.91814        10     
reco:crttag:SingleCRTMatchingProducer               0.00561162     0.0077541     0.0109868    0.00753011    0.00137283       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00439106     0.0062941    0.00926128    0.00621343    0.00117536       10     
reco:pandora2Track:LArPandoraTrackCreation            2.97611       5.82786       7.66976       5.76693       1.41321        10     
reco:pandora2calo:ShowerCalorimetry                  0.870726       1.90991       2.8577        1.93671       0.58869        10     
reco:pandora2calonosce:ShowerCalorimetry              0.81964       1.86191       2.76453       1.90683      0.563165        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.046373      0.0860037     0.0999145     0.0919255     0.0149846       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0460497     0.0807755     0.098635      0.0833179     0.0141884       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.27327       2.52098       3.57956       2.37622      0.684063        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.29264       2.61268       3.76701       2.4697       0.740865        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.26372       2.47963       3.62627       2.41478      0.644135        10     
[art]:TriggerResults:TriggerResultInserter           3.167e-05    5.84485e-05   0.000106301   5.1898e-05    2.30277e-05      10     
end_path:out1:RootOutput                            1.1511e-05    1.71745e-05   3.4726e-05    1.41725e-05   6.70591e-06      10     
end_path:out1:RootOutput(write)                       2.49289       3.33759       4.02031       3.37475      0.408619        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6399.89 MB
  Peak resident set size usage (VmHWM): 4156.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11346632
-rw-r--r-- 1 slot1_7 slot1_7      39221 Nov  3 14:22 jobscript.log
-rw-r--r-- 1 slot1_7 slot1_7        202 Nov  3 14:21 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_7 slot1_7        519 Nov  3 14:21 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_7 slot1_7       3574 Nov  3 14:21 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_7 slot1_7     200704 Nov  3 14:21 mem.db
-rw-r--r-- 1 slot1_7 slot1_7      36864 Nov  3 14:21 time.db
-rw-r--r-- 1 slot1_7 slot1_7 5953892229 Nov  3 14:21 pdhd_prod_beam__231896_144_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_156_1_20251010T171938Z_PDInt.root_236804_29_1_20251103T185346Z_detsim_reco1.root
-rw-r--r-- 1 slot1_7 slot1_7   51590460 Nov  3 14:20 Pandora_Events.pndr
-rw-r--r-- 1 slot1_7 slot1_7          0 Nov  3 13:55 ProtoDUNE.log
-rw-r--r-- 1 slot1_7 slot1_7          0 Nov  3 13:52 debugprod.log
-rw-r--r-- 1 slot1_7 slot1_7        519 Nov  3 13:51 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 slot1_7 slot1_7 5613143013 Nov  3 13:51 temp_detsim.root
-rw-r--r-- 1 slot1_7 slot1_7        123 Nov  3 12:53 all-input-dids.txt
justIN time: 2025-11-05 07:26:22 UTC       justIN version: 01.05.01