justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236917.5@dunegpschedd02.fnal.gov

Jobsub ID236917.5@dunegpschedd02.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 19:55:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-03 22:23:02
From worker nodeHostnamedunegli-7474314-0-fnpc17108.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 20:22:02
Input filesusertests:pdhd_prod_beam__226470_21_1_20251008T212459Z_gen_g4_IonScintPDExt.root_227040_157_1_20251010T172003Z_PDInt.root
JobscriptExit code0
Real time1h (7196s)
CPU time1h (7055s = 98%)
Max RSS bytes6493319168 (6192 MiB)
Outputting started2025-11-03 22:21:58
Output filesusertests:pdhd_prod_beam__226470_21_1_20251008T212459Z_gen_g4_IonScintPDExt.root_227040_157_1_20251010T172003Z_PDInt.root_236917_5_1_20251103T202204Z_detsim_reco1.root
Finished2025-11-03 22:23:02
Saved logsjustin-logs:236917.5-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

4.02549e+07
0 3.71268e+07
1 3.68922e+07
2 3.68574e+07
Now with regularization...
Begin: 3.50329e+07
0 3.50227e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 03-Nov-2025 22:15:31 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13895!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 91114 traces, 91114 summary
Retagger: tagging trace set: gauss with 72321 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72321 traces tagged "gauss"
FrameSaver: q=2.36885e+06 n=1182786 tag=gauss
wclsFrameSaver: saving 91114 traces tagged "wiener"
FrameSaver: q=2.97554e+06 n=1163719 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10810 XUs and 7446 XVs -> 111 XUVs
C:0 T:1 79529 XUs and 37614 XVs -> 1718 XUVs
C:0 T:2 12448 XUs and 61722 XVs -> 465 XUVs
C:0 T:4 71 XUs and 25 XVs -> 0 XUVs
C:0 T:5 13133 XUs and 19287 XVs -> 1879 XUVs
C:0 T:6 81009 XUs and 104363 XVs -> 2585 XUVs
C:0 T:7 19 XUs and 14 XVs -> 1 XUVs
6759 XUVs total
2734 collection wire objects
6759 potential space points
Neighbour search...
301129 tests to find 148702 neighbours
Iterating with no regularization...
Begin: 7.21188e+07
0 6.12459e+07
1 6.07343e+07
2 6.06889e+07
Now with regularization...
Begin: 5.81263e+07
0 5.81055e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 22:18:59 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17008!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 98825 traces, 98825 summary
Retagger: tagging trace set: gauss with 78793 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 78793 traces tagged "gauss"
FrameSaver: q=2.44492e+06 n=1225360 tag=gauss
wclsFrameSaver: saving 98825 traces tagged "wiener"
FrameSaver: q=3.08198e+06 n=1187525 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 15316 XUs and 9235 XVs -> 88 XUVs
C:0 T:1 74299 XUs and 41729 XVs -> 1062 XUVs
C:0 T:2 21083 XUs and 14271 XVs -> 506 XUVs
C:0 T:3 16 XUs and 11 XVs -> 0 XUVs
C:0 T:5 25744 XUs and 37647 XVs -> 853 XUVs
C:0 T:6 22630 XUs and 17328 XVs -> 1647 XUVs
C:0 T:7 10 XUs and 4 XVs -> 0 XUVs
4156 XUVs total
2507 collection wire objects
4156 potential space points
Neighbour search...
111156 tests to find 60602 neighbours
Iterating with no regularization...
Begin: 3.43395e+07
0 3.26935e+07
1 3.25934e+07
2 3.2578e+07
Now with regularization...
Begin: 3.01184e+07
0 3.01101e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 22:21:56 UTC  Closed output file "pdhd_prod_beam__226470_21_1_20251008T212459Z_gen_g4_IonScintPDExt.root_227040_157_1_20251010T172003Z_PDInt.root_236917_5_1_20251103T202204Z_detsim_reco1.root"
03-Nov-2025 22:21:56 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            159.52        184.315       244.389       174.203       26.3779        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000842753   0.00574126     0.0101318    0.00556799    0.00310323       10     
reco:opdec:Deconvolution                             0.982922       1.26827       1.68664       1.2481       0.207678        10     
reco:ophitspe:OpHitFinderDeco                        0.233234      0.400757       1.32023      0.299242      0.312944        10     
reco:opflash:OpFlashFinder                           0.0833215     0.113431       0.14942      0.111524      0.0217212       10     
reco:opslicer:OpSlicer                               0.251769      0.330101      0.476027      0.303897      0.0670811       10     
reco:wclsdatahd:WireCellToolkit                       63.9296       69.6083       80.8209       67.3238       5.80912        10     
reco:gaushit:GausHitFinder                           0.829903       1.0952        1.40938       1.04244      0.189215        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000438093   0.000552335   0.000689848   0.000535502   8.65194e-05      10     
reco:reco3d:SpacePointSolver                         0.698015        1.19         2.09204       1.04365      0.380574        10     
reco:hitpdune:DisambigFromSpacePoints                0.651734      0.930597       1.55123      0.871943      0.290647        10     
reco:pandora:StandardPandora                          21.4974       29.334        37.0493       28.7223       5.1086         10     
reco:pandoraWriter:StandardPandora                   0.265707      0.360037       0.49151      0.349456      0.0799144       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.56392       2.68535       4.5646        2.53971      0.854887        10     
reco:pandoraShower:LArPandoraShowerCreation           1.72587       2.55304       4.82674       2.28818      0.904652        10     
reco:pandoracalo:Calorimetry                         0.791288       1.21674       2.1054        1.16802      0.366385        10     
reco:pandoracalonosce:Calorimetry                    0.859664       1.25771       2.24356       1.16097       0.38664        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.66121       2.3322        4.9955        1.96276      0.955455        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.55334       2.21632       4.17692       1.93844       0.73297        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0512663     0.0675653     0.0907012     0.0635703     0.0134217       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.052863      0.0649348     0.0873829     0.0634364     0.0115853       10     
reco:emtrkmichelid:EmTrackMichelId                    28.0504       38.9006       51.6169       36.2433       7.63166        10     
reco:crttag:SingleCRTMatchingProducer               0.00876945     0.0128844     0.0201266     0.0118949    0.00356015       10     
reco:crtreco:TwoCRTMatchingProducer                  0.0057464    0.00980111     0.0196446    0.00854796    0.00396583       10     
reco:pandora2Track:LArPandoraTrackCreation            5.27919       9.48753       22.7076       7.38712       5.02887        10     
reco:pandora2calo:ShowerCalorimetry                   1.64301       2.50961       5.54578       2.03047       1.19323        10     
reco:pandora2calonosce:ShowerCalorimetry              1.68028       2.46371       4.8311        2.06906      0.992074        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0486213     0.0826161     0.143135      0.0748765     0.0319263       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0496966     0.0841375     0.144098      0.0765754     0.0313094       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.29421       3.6378        7.08683       3.12537       1.45081        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.07574       3.43466       6.70035       2.9729         1.457         10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.00854       3.46555       7.11302       2.88983       1.59303        10     
[art]:TriggerResults:TriggerResultInserter          2.3237e-05    4.18026e-05   0.000100637   3.69065e-05   2.16734e-05      10     
end_path:out1:RootOutput                             6.663e-06    1.09002e-05   3.2075e-05    8.6005e-06    7.26365e-06      10     
end_path:out1:RootOutput(write)                       2.27054       3.07266       4.38458       2.64492      0.846299        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6250.09 MB
  Peak resident set size usage (VmHWM): 4052.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11447368
-rw-r--r-- 1 dunegli fnalgrid      38951 Nov  3 22:21 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Nov  3 22:21 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  3 22:21 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Nov  3 22:21 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Nov  3 22:21 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Nov  3 22:21 time.db
-rw-r--r-- 1 dunegli fnalgrid 6005574414 Nov  3 22:21 pdhd_prod_beam__226470_21_1_20251008T212459Z_gen_g4_IonScintPDExt.root_227040_157_1_20251010T172003Z_PDInt.root_236917_5_1_20251103T202204Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   51829710 Nov  3 22:20 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Nov  3 21:51 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Nov  3 21:48 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  3 21:48 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5664375034 Nov  3 21:48 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Nov  3 20:22 all-input-dids.txt
justIN time: 2025-11-04 16:00:45 UTC       justIN version: 01.05.01