justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237924.1@dunegpschedd02.fnal.gov

Jobsub ID237924.1@dunegpschedd02.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-05 17:53:39
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_gk02_condce_mcore
Last heartbeat2025-11-05 19:52:07
From worker nodeHostnameuct2-c652.mwt2.org
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 18:03:06
Input filesusertests:pdhd_prod_beam__226470_68_1_20251008T212513Z_gen_g4_IonScintPDExt.root_227040_29_1_20251010T170039Z_PDInt.root
JobscriptExit code0
Real time1h (5873s)
CPU time1h (5680s = 96%)
Max RSS bytes7449976832 (7104 MiB)
Outputting started2025-11-05 19:41:00
Output filesusertests:pdhd_prod_beam__226470_68_1_20251008T212513Z_gen_g4_IonScintPDExt.root_227040_29_1_20251010T170039Z_PDInt.root_237924_1_1_20251105T180311Z_detsim_reco1.root
Finished2025-11-05 19:52:07
Saved logsjustin-logs:237924.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ion wire objects
2846 potential space points
Neighbour search...
41684 tests to find 21358 neighbours
Iterating with no regularization...
Begin: 2.81917e+07
0 2.69429e+07
1 2.68666e+07
2 2.68622e+07
Now with regularization...
Begin: 2.56137e+07
0 2.5612e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 05-Nov-2025 13:36:35 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 16098!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 100472 traces, 100472 summary
Retagger: tagging trace set: gauss with 82820 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 82820 traces tagged "gauss"
FrameSaver: q=2.61886e+06 n=1338003 tag=gauss
wclsFrameSaver: saving 100472 traces tagged "wiener"
FrameSaver: q=3.39976e+06 n=1323756 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 18298 XUs and 12928 XVs -> 198 XUVs
C:0 T:1 131256 XUs and 96601 XVs -> 4732 XUVs
C:0 T:2 36568 XUs and 44477 XVs -> 1415 XUVs
C:0 T:3 1 XUs and 10 XVs -> 0 XUVs
C:0 T:5 34057 XUs and 31503 XVs -> 1231 XUVs
C:0 T:6 12293 XUs and 70693 XVs -> 514 XUVs
C:0 T:7 23 XUs and 66 XVs -> 0 XUVs
8090 XUVs total
3521 collection wire objects
8090 potential space points
Neighbour search...
388686 tests to find 149982 neighbours
Iterating with no regularization...
Begin: 6.9977e+07
0 6.25697e+07
1 6.19491e+07
2 6.19245e+07
Now with regularization...
Begin: 6.00442e+07
0 6.00361e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 05-Nov-2025 13:38:56 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 18103!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 102219 traces, 102219 summary
Retagger: tagging trace set: gauss with 79738 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 79738 traces tagged "gauss"
FrameSaver: q=2.51542e+06 n=1258329 tag=gauss
wclsFrameSaver: saving 102219 traces tagged "wiener"
FrameSaver: q=3.10709e+06 n=1199269 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10949 XUs and 9289 XVs -> 65 XUVs
C:0 T:1 58709 XUs and 51126 XVs -> 1578 XUVs
C:0 T:2 44292 XUs and 36893 XVs -> 1827 XUVs
C:0 T:3 28 XUs and 34 XVs -> 1 XUVs
C:0 T:5 8936 XUs and 8113 XVs -> 129 XUVs
C:0 T:6 20535 XUs and 23881 XVs -> 917 XUVs
C:0 T:7 37 XUs and 25 XVs -> 2 XUVs
4519 XUVs total
2257 collection wire objects
4519 potential space points
Neighbour search...
73501 tests to find 39784 neighbours
Iterating with no regularization...
Begin: 3.9656e+07
0 3.71692e+07
1 3.69815e+07
2 3.69674e+07
Now with regularization...
Begin: 3.45766e+07
0 3.45726e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
05-Nov-2025 13:40:59 CST  Closed output file "pdhd_prod_beam__226470_68_1_20251008T212513Z_gen_g4_IonScintPDExt.root_227040_29_1_20251010T170039Z_PDInt.root_237924_1_1_20251105T180311Z_detsim_reco1.root"
05-Nov-2025 13:40:59 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            86.985        122.02        176.383       120.998       27.2099        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000591008   0.00499035     0.0087849    0.00571863    0.00247737       10     
reco:opdec:Deconvolution                             0.732537      0.869569       1.09033      0.838724      0.112101        10     
reco:ophitspe:OpHitFinderDeco                        0.133146      0.141637      0.150182      0.142033     0.00497289       10     
reco:opflash:OpFlashFinder                            0.05521      0.0889158     0.137996      0.0873524     0.0227493       10     
reco:opslicer:OpSlicer                               0.165029      0.234883      0.311885      0.224315      0.0465217       10     
reco:wclsdatahd:WireCellToolkit                       40.219        41.5141       42.172        41.5155      0.541499        10     
reco:gaushit:GausHitFinder                           0.487027      0.718146      0.945172      0.701368      0.144299        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000305863   0.00046798    0.00081619    0.000441417   0.000131645      10     
reco:reco3d:SpacePointSolver                          0.41667      0.884224       2.07953      0.754618      0.504574        10     
reco:hitpdune:DisambigFromSpacePoints                0.332616      0.754646       1.33789      0.721987      0.322723        10     
reco:pandora:StandardPandora                          7.94366       23.5306       56.8552       20.9519       14.3659        10     
reco:pandoraWriter:StandardPandora                   0.197746      0.319287      0.423852      0.320465      0.0703918       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.754939       1.79562       3.1472        1.88293      0.717711        10     
reco:pandoraShower:LArPandoraShowerCreation          0.779767       1.76207       3.03775       1.6688       0.724837        10     
reco:pandoracalo:Calorimetry                          0.40523      0.838507       1.34666      0.878775      0.321498        10     
reco:pandoracalonosce:Calorimetry                    0.401453      0.800288       1.3011       0.843484       0.29975        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.702875       1.63129       2.90675       1.49546      0.676953        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.718373       1.55473       2.64267       1.39155      0.623069        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0391445     0.0546365     0.0741691     0.0531133    0.00959531       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0405724      0.05574      0.0860904     0.0517272     0.0124344       10     
reco:emtrkmichelid:EmTrackMichelId                    21.2908       28.4477       34.3552       29.3344       4.4556         10     
reco:crttag:SingleCRTMatchingProducer               0.00513141    0.00620754    0.00747763    0.00621271    0.000785679      10     
reco:crtreco:TwoCRTMatchingProducer                  0.0035486    0.00536114    0.00752001    0.00505112    0.00123273       10     
reco:pandora2Track:LArPandoraTrackCreation            2.33394       4.70297       8.03058       4.44651       1.81203        10     
reco:pandora2calo:ShowerCalorimetry                  0.674745       1.38523       2.43004       1.27404      0.552985        10     
reco:pandora2calonosce:ShowerCalorimetry             0.662067       1.43779       2.41826       1.34439      0.528354        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0411981     0.0547013     0.081171      0.052408      0.0101713       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.038351      0.0527232     0.0785881     0.0520058     0.0106415       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.993166       2.03132       3.25419       1.98648      0.766145        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.00629       2.01346       3.38741       1.95352      0.778722        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.02701       1.88371       3.16856       1.86348      0.704476        10     
[art]:TriggerResults:TriggerResultInserter          2.0258e-05    2.9888e-05    8.5711e-05    2.3544e-05    1.87781e-05      10     
end_path:out1:RootOutput                             7.815e-06    1.06802e-05   2.4657e-05    9.2325e-06    4.71527e-06      10     
end_path:out1:RootOutput(write)                       1.85454       2.32417       2.68655       2.32518      0.252336        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6314.66 MB
  Peak resident set size usage (VmHWM): 4080.89 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12564456
-rw-r--r--. 1 dune osgvo      39583 Nov  5 13:41 jobscript.log
-rw-r--r--. 1 dune osgvo        200 Nov  5 13:41 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Nov  5 13:40 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Nov  5 13:40 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     200704 Nov  5 13:40 mem.db
-rw-r--r--. 1 dune osgvo      36864 Nov  5 13:40 time.db
-rw-r--r--. 1 dune osgvo 6586731522 Nov  5 13:40 pdhd_prod_beam__226470_68_1_20251008T212513Z_gen_g4_IonScintPDExt.root_227040_29_1_20251010T170039Z_PDInt.root_237924_1_1_20251105T180311Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   55236960 Nov  5 13:40 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Nov  5 13:20 ProtoDUNE.log
-rw-r--r--. 1 dune osgvo          0 Nov  5 13:18 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Nov  5 13:18 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6223708831 Nov  5 13:18 temp_detsim.root
-rw-r--r--. 1 dune osgvo        121 Nov  5 12:03 all-input-dids.txt
justIN time: 2026-02-11 12:16:27 UTC       justIN version: 01.06.00