justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239443.1@dunegpschedd01.fnal.gov

Jobsub ID239443.1@dunegpschedd01.fnal.gov
Workflow ID9383
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-28 18:07:27
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-10-28 19:07:52
From worker nodeHostnamedunegli-7387498-0-fnpc22001.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-28 18:08:07
Input filesusertests:pdhd_prod_beam__226698_63_1_20251009T171411Z_gen_g4_IonScintPDExt.root_226750_154_1_20251009T212121Z_PDInt.root
JobscriptExit code0
Real time58m (3533s)
CPU time57m (3450s = 97%)
Max RSS bytes6785204224 (6470 MiB)
Outputting started2025-10-28 19:07:00
Output filesusertests:pdhd_prod_beam__226698_63_1_20251009T171411Z_gen_g4_IonScintPDExt.root_226750_154_1_20251009T212121Z_PDInt.root_239443_1_1_20251028T180809Z_detsim_reco1.root
Finished2025-10-28 19:07:52
Saved logsjustin-logs:239443.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000160515 0.000160515 0.000160545 0.000160515 0.000160515 0.000160515 0.000160515 0.000160515 0.000160515 0.000160515 
----view 1: 0.0291591 0.0429288 0.0328503 0.0243443 0.0310866 0.0188457 0.0127927 0.0381282 0.0255985 0.0249009 0.0231869 0.0310257 0.0282955 
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 9th record. run: 20250627 subRun: 1 event: 78 at 28-Oct-2025 19:04:48 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14475!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 74116 traces, 74116 summary
Retagger: tagging trace set: gauss with 60739 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 60739 traces tagged "gauss"
FrameSaver: q=1.57525e+06 n=855574 tag=gauss
wclsFrameSaver: saving 74116 traces tagged "wiener"
FrameSaver: q=2.0624e+06 n=847525 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7184 XUs and 7154 XVs -> 69 XUVs
C:0 T:1 48483 XUs and 57458 XVs -> 2226 XUVs
C:0 T:2 3370 XUs and 3494 XVs -> 367 XUVs
C:0 T:5 2074 XUs and 5040 XVs -> 180 XUVs
C:0 T:6 37166 XUs and 30871 XVs -> 1613 XUVs
C:0 T:7 51 XUs and 17 XVs -> 0 XUVs
4455 XUVs total
1966 collection wire objects
4455 potential space points
Neighbour search...
202757 tests to find 75678 neighbours
Iterating with no regularization...
Begin: 2.53339e+07
0 2.24585e+07
1 2.22143e+07
2 2.21902e+07
3 2.2185e+07
Now with regularization...
Begin: 2.09852e+07
0 2.09822e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 79 at 28-Oct-2025 19:05:52 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14322!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 76065 traces, 76065 summary
Retagger: tagging trace set: gauss with 61415 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 61415 traces tagged "gauss"
FrameSaver: q=1.71584e+06 n=928209 tag=gauss
wclsFrameSaver: saving 76065 traces tagged "wiener"
FrameSaver: q=2.09632e+06 n=900898 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7214 XUs and 6119 XVs -> 101 XUVs
C:0 T:1 35062 XUs and 32200 XVs -> 837 XUVs
C:0 T:2 11687 XUs and 16894 XVs -> 672 XUVs
C:0 T:3 0 XUs and 1 XVs -> 0 XUVs
C:0 T:4 6 XUs and 12 XVs -> 4 XUVs
C:0 T:5 8809 XUs and 13200 XVs -> 2382 XUVs
C:0 T:6 11407 XUs and 10499 XVs -> 652 XUVs
C:0 T:7 10 XUs and 5 XVs -> 0 XUVs
4648 XUVs total
1940 collection wire objects
4648 potential space points
Neighbour search...
141390 tests to find 84360 neighbours
Iterating with no regularization...
Begin: 2.19041e+07
0 2.02092e+07
1 2.00334e+07
2 2.00088e+07
3 2.00022e+07
Now with regularization...
Begin: 1.81593e+07
0 1.81555e+07
28-Oct-2025 19:06:59 UTC  Closed output file "pdhd_prod_beam__226698_63_1_20251009T171411Z_gen_g4_IonScintPDExt.root_226750_154_1_20251009T212121Z_PDInt.root_239443_1_1_20251028T180809Z_detsim_reco1.root"
28-Oct-2025 19:06:59 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            62.0641       85.661        120.808       80.0478       20.4354        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000540456   0.00406937    0.00665015    0.00474748    0.00198989       10     
reco:opdec:Deconvolution                             0.445021      0.565184      0.710975      0.543789      0.0823392       10     
reco:ophitspe:OpHitFinderDeco                        0.0971642     0.103426      0.109701      0.102262     0.00401095       10     
reco:opflash:OpFlashFinder                           0.041284      0.0670587     0.094016      0.0657579     0.0175527       10     
reco:opslicer:OpSlicer                               0.111187      0.166171      0.228636      0.165764      0.0319347       10     
reco:wclsdatahd:WireCellToolkit                       31.1605       31.6561       32.3183       31.6724      0.368451        10     
reco:gaushit:GausHitFinder                           0.323984      0.480508      0.653412      0.457145      0.105374        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000143883   0.000242808   0.000444475   0.000199329   0.000103735      10     
reco:reco3d:SpacePointSolver                          0.24359      0.799468       1.95396      0.750199       0.50452        10     
reco:hitpdune:DisambigFromSpacePoints                0.202899      0.448788      0.833219      0.338433      0.220243        10     
reco:pandora:StandardPandora                          6.46407       16.1474       35.1505       13.4491       9.1488         10     
reco:pandoraWriter:StandardPandora                   0.0937947     0.160342      0.232494      0.149042      0.0411275       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.528859      0.990259       1.8002       0.910026      0.402262        10     
reco:pandoraShower:LArPandoraShowerCreation          0.530725      0.977743       2.10337      0.722253      0.480131        10     
reco:pandoracalo:Calorimetry                         0.298375      0.544982       1.04806      0.481985      0.230495        10     
reco:pandoracalonosce:Calorimetry                    0.297437      0.536398       1.03114      0.472833      0.225638        10     
reco:pandoraShowercalo:ShowerCalorimetry              0.54815       1.03337       2.39201      0.730603      0.560401        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         0.52276      0.987141       2.24975      0.697858      0.530783        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0210204     0.0299948     0.0428891     0.0299178    0.00714232       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0208466     0.0298957     0.0422312     0.0300608    0.00697504       10     
reco:emtrkmichelid:EmTrackMichelId                    14.5557       20.5187       27.6081       19.5881       4.32505        10     
reco:crttag:SingleCRTMatchingProducer               0.00210519     0.0037062    0.00571801    0.00348794    0.00103452       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00176843    0.00277013    0.00373334    0.00274968    0.00061437       10     
reco:pandora2Track:LArPandoraTrackCreation            1.28934       2.48281       5.18999       1.97558       1.20716        10     
reco:pandora2calo:ShowerCalorimetry                  0.465668      0.914831       2.08207      0.648571      0.501912        10     
reco:pandora2calonosce:ShowerCalorimetry             0.456911      0.893127       2.04515      0.631114      0.492792        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0213689     0.0297959     0.0419577     0.029774     0.00672617       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0205559     0.0293686     0.0427115     0.029914      0.0068427       10     
reco:pandora2Shower:LArPandoraShowerCreation          0.58552       1.11415       2.26799      0.885995      0.519106        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.634426       1.23414       2.67522      0.944524      0.635758        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.620501       1.20205       2.58818      0.926496       0.61504        10     
[art]:TriggerResults:TriggerResultInserter          1.6552e-05    2.41578e-05   5.7499e-05    2.06385e-05   1.12354e-05      10     
end_path:out1:RootOutput                             3.126e-06    6.8138e-06    1.9847e-05     5.941e-06    4.47131e-06      10     
end_path:out1:RootOutput(write)                       1.10685       1.43255       2.01526       1.35512      0.283252        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6157.6 MB
  Peak resident set size usage (VmHWM): 3953.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11273300
-rw-r--r-- 1 dunegli fnalgrid      40710 Oct 28 19:07 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 28 19:07 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 28 19:07 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 28 19:07 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 28 19:07 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 28 19:06 time.db
-rw-r--r-- 1 dunegli fnalgrid 5914557422 Oct 28 19:06 pdhd_prod_beam__226698_63_1_20251009T171411Z_gen_g4_IonScintPDExt.root_226750_154_1_20251009T212121Z_PDInt.root_239443_1_1_20251028T180809Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   51013510 Oct 28 19:06 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 28 18:53 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 28 18:51 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 28 18:51 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5577983931 Oct 28 18:51 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 28 18:08 all-input-dids.txt
justIN time: 2025-11-05 21:05:18 UTC       justIN version: 01.05.01