justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 238689.0@dunegpschedd01.fnal.gov

Jobsub ID238689.0@dunegpschedd01.fnal.gov
Workflow ID9360
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-26 21:35:09
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-10-27 01:42:32
From worker nodeHostnamemh-7763-11.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-26 21:46:07
Input filesmonte-carlo-009360-000001
JobscriptExit code0
Real time3h (14036s)
CPU time3h (13731s = 97%)
Max RSS bytes4286459904 (4087 MiB)
Outputting started2025-10-27 01:40:03
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09360/1/001/reco_stage1.root
Finished2025-10-27 01:42:32
Saved logsjustin-logs:238689.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 194477 traces, 0 summary
Retagger: tagging trace set: gauss with 147838 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 147838 traces tagged "gauss"
FrameSaver: q=5.23694e+06 n=2642688 tag=gauss
wclsFrameSaver: saving 194477 traces tagged "wiener"
FrameSaver: q=5.52522e+06 n=2326108 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 45248 XUs and 56390 XVs -> 3601 XUVs
C:0 T:1 76463 XUs and 69021 XVs -> 4527 XUVs
C:0 T:2 206956 XUs and 208219 XVs -> 16487 XUVs
C:0 T:3 73767 XUs and 72497 XVs -> 984 XUVs
C:0 T:4 11841 XUs and 12358 XVs -> 129 XUVs
C:0 T:5 56456 XUs and 59515 XVs -> 6443 XUVs
C:0 T:6 253618 XUs and 239246 XVs -> 11079 XUVs
C:0 T:7 83810 XUs and 78162 XVs -> 1139 XUVs
C:0 T:8 13929 XUs and 13481 XVs -> 574 XUVs
C:0 T:9 105331 XUs and 150160 XVs -> 12937 XUVs
C:0 T:10 51144 XUs and 39504 XVs -> 4128 XUVs
C:0 T:11 58 XUs and 52 XVs -> 7 XUVs
62035 XUVs total
19312 collection wire objects
54904 potential space points
Neighbour search...
5826830 tests to find 2715972 neighbours
Iterating with no regularization...
Begin: 1.20772e+08
0 1.00131e+08
1 9.87324e+07
2 9.85523e+07
3 9.85036e+07
Now with regularization...
Begin: 8.31698e+07
0 8.31207e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410134513
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410134513
fLifetime: 30000 [us]
26-Oct-2025 18:40:00 PDT  Closed output file "reco_stage1.root"
26-Oct-2025 18:40:00 PDT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          294.938       435.374       684.444       405.366       106.302        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00118782    0.00396828    0.00621656    0.00404703    0.00165554       10     
reco:ophit:OpHitFinder                             0.269205       0.3731       0.445409      0.383657      0.0598679       10     
reco:opflash:OpFlashFinder                         0.161856      0.217792      0.274479      0.209454      0.0351011       10     
reco:opslicer:OpSlicer                             0.529028       0.85761       1.09468      0.878473      0.184052        10     
reco:caldata:DataPrepModule                         4.71726       6.62526       8.18898       6.91883       1.33549        10     
reco:wclsdatasp:WireCellToolkit                     31.779        39.9499       43.6719       41.0975       3.82328        10     
reco:gaushit:GausHitFinder                         0.815149       1.48107       2.08023       1.55874      0.400475        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000435193   0.000613617   0.000786438   0.00062282    0.000117787      10     
reco:reco3d:SpacePointSolver                        7.41047       20.6505       33.7858       20.9884       7.06933        10     
reco:hitpdune:DisambigFromSpacePoints               1.05246       1.95306       3.40271       1.47546      0.857198        10     
reco:pandora:StandardPandora                        17.502        46.0388       86.3964       41.9511       21.3346        10     
reco:pandoraWriter:StandardPandora                 0.364733      0.526903      0.730308      0.508736      0.115006        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.41394       5.4887        12.3307       4.91761       2.4425         10     
reco:pandoraShower:LArPandoraShowerCreation         2.88239       6.48694       16.9001       5.64768       3.77256        10     
reco:pandoracalo:Calorimetry                        1.88176       3.1343        5.95268       3.01912       1.19856        10     
reco:pandoracalonosce:Calorimetry                   1.7764        2.92036       6.46649       2.55557       1.3393         10     
reco:pandorapid:Chi2ParticleID                    0.00475359    0.00752358     0.0120899    0.00773201    0.00214606       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0385996     0.0756676     0.125314      0.0793378     0.0265603       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00486364    0.00755424      0.01188     0.00796629     0.0022498       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.0394        6.67763       16.8778       5.45156       3.76843        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.45569       6.64053       17.9317       5.57444        4.109         10     
reco:emtrkmichelid:EmTrackMichelId                  168.017       226.201       320.062       216.158       41.365         10     
reco:crttag:SingleCRTMatchingProducer              0.0153246     0.022347      0.0293181     0.023359     0.00526075       10     
reco:crtreco:TwoCRTMatchingProducer               0.00796121     0.0120341     0.0167668     0.0114647    0.00292283       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.960482       1.20262       1.67919       1.17725      0.199693        10     
reco:pandora2Track:LArPandoraTrackCreation          7.91439       17.4112       37.163        17.3318       7.99883        10     
reco:pandora2calo:Calorimetry                       3.66957       8.60648       18.0741       7.94614       3.89525        10     
reco:pandora2calonosce:Calorimetry                  4.41106       7.97188       14.1745       8.03785       2.89862        10     
reco:pandora2pid:Chi2ParticleID                    0.0103418     0.0211661     0.0334985     0.0208655    0.00582042       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0538512     0.108551      0.161089      0.112286      0.0279804       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0106264     0.0212774     0.0341436     0.0204843    0.00591554       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.96375       6.13981       11.756        5.6644        2.13394        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.23753       6.93639       10.6847       6.17757       1.88096        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.99126       6.65329       11.4883       6.42068       2.01213        10     
[art]:TriggerResults:TriggerResultInserter         6.729e-05    8.67467e-05   0.000132268   7.84135e-05   2.14281e-05      10     
end_path:out1:RootOutput                           1.147e-05    1.68505e-05   1.9149e-05    1.80395e-05   2.40208e-06      10     
end_path:out1:RootOutput(write)                     2.57569       3.80144       5.72242       3.72565      0.830144        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5181.37 MB
  Peak resident set size usage (VmHWM): 4286.46 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.30 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.16 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10692852
-rw-r--r-- 1 cuser cuser 2490078830 Oct 26 17:24 detsim_stage2.root
-rw-r--r-- 1 cuser cuser 2366004488 Oct 26 17:22 detsim_stage1.root
-rw-r--r-- 1 cuser cuser 2200680449 Oct 26 18:40 reco_stage1.root
-rw-r--r-- 1 cuser cuser 1941401025 Oct 26 15:08 g4_stage2.root
-rw-r--r-- 1 cuser cuser 1869765955 Oct 26 14:55 g4_stage1.root
-rw-r--r-- 1 cuser cuser   80917340 Oct 26 18:31 Pandora_Events.pndr
-rw-r--r-- 1 cuser cuser     243120 Oct 26 18:40 jobscript.log
-rw-r--r-- 1 cuser cuser     217088 Oct 26 18:40 mem.db
-rw-r--r-- 1 cuser cuser      62376 Oct 26 18:40 Pandora_Geometry.xml
-rw-r--r-- 1 cuser cuser      36864 Oct 26 18:40 time.db
-rw-r--r-- 1 cuser cuser        519 Oct 26 17:24 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 cuser cuser        519 Oct 26 15:08 g4_protoDUNE_hist.root
-rw-r--r-- 1 cuser cuser        519 Oct 26 18:40 protoDUNE_reco1_hist.root
-rw-r--r-- 1 cuser cuser         52 Oct 26 14:46 all-input-dids.txt
-rw-r--r-- 1 cuser cuser          7 Oct 26 18:40 justin-processed-pfns.txt
-rw-r--r-- 1 cuser cuser          0 Oct 26 17:31 ProtoDUNE.log
-rw-r--r-- 1 cuser cuser          0 Oct 26 17:25 debugprod.log
justIN time: 2025-11-04 07:30:38 UTC       justIN version: 01.05.01