justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242633.2@dunegpschedd01.fnal.gov

Jobsub ID242633.2@dunegpschedd01.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 20:33:15
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_condce_mcore
Last heartbeat2025-11-03 22:31:06
From worker nodeHostnamemwt2-c019.campuscluster.illinois.edu
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 20:34:07
Input filesusertests:pdhd_prod_beam__231896_122_1_20251008T212353Z_gen_g4_IonScintPDExt.root_227040_57_1_20251010T170051Z_PDInt.root
JobscriptExit code0
Real time1h (6896s)
CPU time1h (6492s = 94%)
Max RSS bytes6924615680 (6603 MiB)
Outputting started2025-11-03 22:29:04
Output filesusertests:pdhd_prod_beam__231896_122_1_20251008T212353Z_gen_g4_IonScintPDExt.root_227040_57_1_20251010T170051Z_PDInt.root_242633_2_1_20251103T203412Z_detsim_reco1.root
Finished2025-11-03 22:31:06
Saved logsjustin-logs:242633.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

collection wire objects
5984 potential space points
Neighbour search...
201260 tests to find 100854 neighbours
Iterating with no regularization...
Begin: 3.01854e+07
0 2.80302e+07
1 2.78646e+07
2 2.78407e+07
Now with regularization...
Begin: 2.63748e+07
0 2.63667e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 03-Nov-2025 16:24:52 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12602!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 80317 traces, 80317 summary
Retagger: tagging trace set: gauss with 64667 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 64667 traces tagged "gauss"
FrameSaver: q=1.7623e+06 n=955161 tag=gauss
wclsFrameSaver: saving 80317 traces tagged "wiener"
FrameSaver: q=2.22774e+06 n=925161 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10630 XUs and 13095 XVs -> 91 XUVs
C:0 T:1 49987 XUs and 58490 XVs -> 1846 XUVs
C:0 T:2 15123 XUs and 16683 XVs -> 896 XUVs
C:0 T:4 21 XUs and 33 XVs -> 0 XUVs
C:0 T:5 19681 XUs and 34016 XVs -> 1113 XUVs
C:0 T:6 13867 XUs and 5682 XVs -> 161 XUVs
4107 XUVs total
2145 collection wire objects
4107 potential space points
Neighbour search...
127033 tests to find 65688 neighbours
Iterating with no regularization...
Begin: 3.19169e+07
0 2.98327e+07
1 2.96935e+07
2 2.96839e+07
Now with regularization...
Begin: 2.76274e+07
0 2.76225e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 16:26:56 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 11893!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 86385 traces, 86385 summary
Retagger: tagging trace set: gauss with 67020 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 67020 traces tagged "gauss"
FrameSaver: q=1.97674e+06 n=968062 tag=gauss
wclsFrameSaver: saving 86385 traces tagged "wiener"
FrameSaver: q=2.609e+06 n=953053 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10170 XUs and 10421 XVs -> 116 XUVs
C:0 T:1 66478 XUs and 68740 XVs -> 2496 XUVs
C:0 T:2 12657 XUs and 10569 XVs -> 250 XUVs
C:0 T:3 5 XUs and 12 XVs -> 0 XUVs
C:0 T:4 7 XUs and 7 XVs -> 0 XUVs
C:0 T:5 32149 XUs and 34922 XVs -> 1805 XUVs
C:0 T:6 26903 XUs and 26461 XVs -> 1218 XUVs
C:0 T:7 0 XUs and 0 XVs -> 0 XUVs
5885 XUVs total
2094 collection wire objects
5885 potential space points
Neighbour search...
237195 tests to find 139706 neighbours
Iterating with no regularization...
Begin: 4.35693e+07
0 3.98129e+07
1 3.94602e+07
2 3.94088e+07
3 3.93942e+07
Now with regularization...
Begin: 3.72716e+07
0 3.72629e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 16:29:02 CST  Closed output file "pdhd_prod_beam__231896_122_1_20251008T212353Z_gen_g4_IonScintPDExt.root_227040_57_1_20251010T170051Z_PDInt.root_242633_2_1_20251103T203412Z_detsim_reco1.root"
03-Nov-2025 16:29:02 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            117.138       152.034       208.101       142.725       32.2226        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00126733    0.00711482     0.0106017    0.00750596    0.00285149       10     
reco:opdec:Deconvolution                             0.769408      0.988937       1.24855       0.97668      0.150367        10     
reco:ophitspe:OpHitFinderDeco                        0.155431      0.218474      0.241199      0.229168      0.0267579       10     
reco:opflash:OpFlashFinder                           0.0670872     0.097082      0.137968      0.0953572     0.0227159       10     
reco:opslicer:OpSlicer                               0.234629      0.303917      0.403548       0.28821      0.057201        10     
reco:wclsdatahd:WireCellToolkit                       44.6338       49.8915       54.842        49.4103       2.81571        10     
reco:gaushit:GausHitFinder                           0.667638      0.874449       1.15502       0.85594      0.141729        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000334929   0.000502741   0.000733287   0.000446835   0.000122169      10     
reco:reco3d:SpacePointSolver                         0.750931       1.27508       2.20746       1.16376      0.415888        10     
reco:hitpdune:DisambigFromSpacePoints                0.455786      0.882608       1.87766       0.67194      0.435954        10     
reco:pandora:StandardPandora                          13.4536       27.1281       48.6938       27.319        10.912         10     
reco:pandoraWriter:StandardPandora                   0.272815      0.375023      0.528588      0.346792      0.0738364       10     
reco:pandoraTrack:LArPandoraTrackCreation             1.41974       2.60363       4.9367        2.03148       1.16987        10     
reco:pandoraShower:LArPandoraShowerCreation          0.931656       2.26677       4.45428       2.03269       1.13445        10     
reco:pandoracalo:Calorimetry                         0.718588       1.22336       2.45419       0.9906       0.546186        10     
reco:pandoracalonosce:Calorimetry                    0.682802       1.07362       1.8575       0.902593      0.404655        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.02309       2.08779       4.18196       1.72999       1.10484        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         0.8496        1.98911       4.91744       1.69954       1.18892        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0442679     0.0672291     0.0916136     0.0644812     0.0151973       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0502708     0.0638668     0.0851624     0.058806      0.0123636       10     
reco:emtrkmichelid:EmTrackMichelId                    28.1988       35.5922       46.9638       35.3716       6.01321        10     
reco:crttag:SingleCRTMatchingProducer                0.0058311    0.00916782     0.0177172    0.00797587    0.00351181       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00542745    0.00731883     0.0115031    0.00665191    0.00188724       10     
reco:pandora2Track:LArPandoraTrackCreation            3.68256       7.52514       13.7494       6.59681       3.5683         10     
reco:pandora2calo:ShowerCalorimetry                  0.880347       2.21935       4.40252       2.11632       1.14601        10     
reco:pandora2calonosce:ShowerCalorimetry             0.841864       2.06788       4.21043       1.60855       1.13009        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0453291     0.0711651     0.109443      0.0597753     0.0236999       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0439485     0.0687904     0.105656      0.058965      0.0218146       10     
reco:pandora2Shower:LArPandoraShowerCreation          1.35594       2.96409       5.85874       2.22295       1.61362        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.22067       2.72963       5.70534       2.28563       1.47296        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.29217       2.53718       4.82257       2.24923       1.20416        10     
[art]:TriggerResults:TriggerResultInserter          2.7822e-05    5.02625e-05   0.000136296   3.42695e-05   3.48051e-05      10     
end_path:out1:RootOutput                             9.568e-06    1.68115e-05   3.6699e-05    1.11455e-05   9.50948e-06      10     
end_path:out1:RootOutput(write)                       1.95395       2.64622       3.57319       2.70354      0.444711        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6094.57 MB
  Peak resident set size usage (VmHWM): 3863.09 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12054872
-rw-r--r-- 1 dune osgvo      39931 Nov  3 16:29 jobscript.log
-rw-r--r-- 1 dune osgvo        201 Nov  3 16:29 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Nov  3 16:29 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Nov  3 16:29 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     200704 Nov  3 16:29 mem.db
-rw-r--r-- 1 dune osgvo      36864 Nov  3 16:29 time.db
-rw-r--r-- 1 dune osgvo 6322685359 Nov  3 16:29 pdhd_prod_beam__231896_122_1_20251008T212353Z_gen_g4_IonScintPDExt.root_227040_57_1_20251010T170051Z_PDInt.root_242633_2_1_20251103T203412Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   53503030 Nov  3 16:28 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Nov  3 16:04 ProtoDUNE.log
-rw-r--r-- 1 dune osgvo          0 Nov  3 16:01 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Nov  3 16:01 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 5967675998 Nov  3 16:01 temp_detsim.root
-rw-r--r-- 1 dune osgvo        122 Nov  3 14:34 all-input-dids.txt
justIN time: 2025-11-04 15:40:54 UTC       justIN version: 01.05.01