justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242592.6@dunegpschedd01.fnal.gov

Jobsub ID242592.6@dunegpschedd01.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 19:59:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-03 22:33:05
From worker nodeHostnamedunegli-7474314-0-fnpc17108.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 20:18:23
Input filesusertests:pdhd_prod_beam__231896_143_1_20251008T212400Z_gen_g4_IonScintPDExt.root_227040_106_1_20251010T170634Z_PDInt.root
JobscriptExit code0
Real time2h (8017s)
CPU time2h (7845s = 97%)
Max RSS bytes7049113600 (6722 MiB)
Outputting started2025-11-03 22:32:01
Output filesusertests:pdhd_prod_beam__231896_143_1_20251008T212400Z_gen_g4_IonScintPDExt.root_227040_106_1_20251010T170634Z_PDInt.root_242592_6_1_20251103T201826Z_detsim_reco1.root
Finished2025-11-03 22:33:05
Saved logsjustin-logs:242592.6-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

10 potential space points
Neighbour search...
248078 tests to find 123344 neighbours
Iterating with no regularization...
Begin: 3.14486e+07
0 2.88858e+07
1 2.87198e+07
2 2.86954e+07
Now with regularization...
Begin: 2.68044e+07
0 2.67944e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 03-Nov-2025 22:26:29 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15026!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 97895 traces, 97895 summary
Retagger: tagging trace set: gauss with 76696 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76696 traces tagged "gauss"
FrameSaver: q=2.41789e+06 n=1189022 tag=gauss
wclsFrameSaver: saving 97895 traces tagged "wiener"
FrameSaver: q=3.19895e+06 n=1148752 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12053 XUs and 10157 XVs -> 184 XUVs
C:0 T:1 95301 XUs and 74866 XVs -> 2766 XUVs
C:0 T:2 46689 XUs and 30939 XVs -> 893 XUVs
C:0 T:4 16 XUs and 17 XVs -> 0 XUVs
C:0 T:5 32728 XUs and 32220 XVs -> 698 XUVs
C:0 T:6 54416 XUs and 40185 XVs -> 1156 XUVs
C:0 T:7 3 XUs and 4 XVs -> 0 XUVs
5697 XUVs total
2308 collection wire objects
5697 potential space points
Neighbour search...
141253 tests to find 71618 neighbours
Iterating with no regularization...
Begin: 4.60121e+07
0 4.36647e+07
1 4.34578e+07
2 4.34418e+07
Now with regularization...
Begin: 4.19965e+07
0 4.1991e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 22:29:31 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14966!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 83179 traces, 83179 summary
Retagger: tagging trace set: gauss with 68096 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68096 traces tagged "gauss"
FrameSaver: q=2.09003e+06 n=1075784 tag=gauss
wclsFrameSaver: saving 83179 traces tagged "wiener"
FrameSaver: q=2.6793e+06 n=1084139 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10566 XUs and 9542 XVs -> 71 XUVs
C:0 T:1 83232 XUs and 64922 XVs -> 2702 XUVs
C:0 T:2 12361 XUs and 14348 XVs -> 784 XUVs
C:0 T:3 7 XUs and 14 XVs -> 2 XUVs
C:0 T:4 2 XUs and 11 XVs -> 0 XUVs
C:0 T:5 16567 XUs and 10740 XVs -> 384 XUVs
C:0 T:6 27499 XUs and 31500 XVs -> 2981 XUVs
C:0 T:7 17 XUs and 27 XVs -> 8 XUVs
6932 XUVs total
2578 collection wire objects
6932 potential space points
Neighbour search...
384044 tests to find 156436 neighbours
Iterating with no regularization...
Begin: 3.74609e+07
0 3.4203e+07
1 3.39754e+07
2 3.39433e+07
Now with regularization...
Begin: 3.1575e+07
0 3.15626e+07
03-Nov-2025 22:31:59 UTC  Closed output file "pdhd_prod_beam__231896_143_1_20251008T212400Z_gen_g4_IonScintPDExt.root_227040_106_1_20251010T170634Z_PDInt.root_242592_6_1_20251103T201826Z_detsim_reco1.root"
03-Nov-2025 22:31:59 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            142.018       194.27        254.474       185.826       40.6759        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00114434    0.00597909      0.00967     0.00625542    0.00300001       10     
reco:opdec:Deconvolution                              0.93385       1.30366       1.83616       1.26006      0.256534        10     
reco:ophitspe:OpHitFinderDeco                        0.222645      0.274545      0.434254      0.234209      0.0763694       10     
reco:opflash:OpFlashFinder                           0.0785085     0.137624      0.191383      0.143933      0.0352634       10     
reco:opslicer:OpSlicer                               0.200746      0.337938       0.50601       0.32414      0.0881059       10     
reco:wclsdatahd:WireCellToolkit                       61.2834       64.243        68.5331       64.333        2.13897        10     
reco:gaushit:GausHitFinder                           0.781913       1.14904       1.55512       1.07728      0.232345        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000430297   0.000560171   0.000728431   0.000564396   9.3256e-05       10     
reco:reco3d:SpacePointSolver                         0.532632       1.20052       1.90016       1.15749      0.442273        10     
reco:hitpdune:DisambigFromSpacePoints                0.642049       1.00748       1.68739      0.884673      0.331613        10     
reco:pandora:StandardPandora                          16.4555       34.3859       52.7536       33.8538       11.7636        10     
reco:pandoraWriter:StandardPandora                   0.272326      0.374637      0.482772      0.380387      0.064795        10     
reco:pandoraTrack:LArPandoraTrackCreation             1.65751       3.59798       6.08649       3.48716       1.46149        10     
reco:pandoraShower:LArPandoraShowerCreation           1.19504       3.4021        5.65711       3.28193       1.41682        10     
reco:pandoracalo:Calorimetry                         0.770163       1.59315       3.00752       1.52693      0.683616        10     
reco:pandoracalonosce:Calorimetry                     0.73398       1.59405       3.13078       1.47254      0.692769        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.1231        3.29964       8.00581       2.87499       1.8611         10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.07329       3.3426        8.59865       2.65185       2.12132        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0437162     0.0734487     0.136803      0.0649952     0.0254722       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0467778     0.0771175     0.138128      0.0668625     0.0253303       10     
reco:emtrkmichelid:EmTrackMichelId                    28.3845       38.2346       55.1277       35.9777       8.96223        10     
reco:crttag:SingleCRTMatchingProducer               0.00787997     0.0139669     0.0257837     0.0137742    0.00470808       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00553481     0.0108479     0.0230492    0.00967529    0.00485342       10     
reco:pandora2Track:LArPandoraTrackCreation            3.77942       11.3256       21.2906       11.4095       5.56604        10     
reco:pandora2calo:ShowerCalorimetry                  0.944691       3.29374       5.97964       3.37686       1.61251        10     
reco:pandora2calonosce:ShowerCalorimetry             0.904044       3.27663       6.03895       3.39321       1.60653        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0487459     0.0808715     0.123474      0.078963      0.0249089       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0465232     0.0813577     0.124456      0.0794023     0.0247057       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.52821       4.59217       7.45771       4.85144       2.0455         10     
reco:pandora2Showercalo:ShowerCalorimetry             1.45741       4.32592       6.83182       4.67499       1.91465        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.42118       4.24524       6.92722       4.61703       1.90261        10     
[art]:TriggerResults:TriggerResultInserter           2.948e-05    4.11273e-05   7.9162e-05    3.77445e-05   1.38545e-05      10     
end_path:out1:RootOutput                             6.961e-06     9.862e-06    2.2666e-05     8.668e-06    4.43442e-06      10     
end_path:out1:RootOutput(write)                       1.9348        3.25116       4.23785       3.3293       0.722852        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6352.24 MB
  Peak resident set size usage (VmHWM): 4149.21 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12091012
-rw-r--r-- 1 dunegli fnalgrid      39032 Nov  3 22:32 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        202 Nov  3 22:32 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  3 22:32 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Nov  3 22:32 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Nov  3 22:32 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Nov  3 22:32 time.db
-rw-r--r-- 1 dunegli fnalgrid 6346288775 Nov  3 22:31 pdhd_prod_beam__231896_143_1_20251008T212400Z_gen_g4_IonScintPDExt.root_227040_106_1_20251010T170634Z_PDInt.root_242592_6_1_20251103T201826Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   55497770 Nov  3 22:31 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Nov  3 21:59 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Nov  3 21:56 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  3 21:56 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5979084270 Nov  3 21:56 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        123 Nov  3 20:18 all-input-dids.txt
justIN time: 2025-11-04 15:55:27 UTC       justIN version: 01.05.01