justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 270842.0@dunegpschedd01.fnal.gov

Jobsub ID270842.0@dunegpschedd01.fnal.gov
Workflow ID11407
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-29 21:07:46
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2025-12-29 23:58:24
From worker nodeHostnamewn-s04-05-07-02.cr.cnaf.infn.it
cpuinfoAMD EPYC 7313 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-29 21:14:02
Input filesmonte-carlo-011407-000001
JobscriptExit code0
Real time2h (9547s)
CPU time2h (9219s = 96%)
Max RSS bytes4041256960 (3854 MiB)
Outputting started2025-12-29 23:53:10
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11407/1/001/reco_stage1.root
Finished2025-12-29 23:58:24
Saved logsjustin-logs:270842.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

vice:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 128464 traces, 0 summary
Retagger: tagging trace set: gauss with 100226 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 100226 traces tagged "gauss"
FrameSaver: q=3.42599e+06 n=1824523 tag=gauss
wclsFrameSaver: saving 128464 traces tagged "wiener"
FrameSaver: q=3.61094e+06 n=1630384 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 21607 XUs and 22394 XVs -> 282 XUVs
C:0 T:1 53223 XUs and 54027 XVs -> 5925 XUVs
C:0 T:2 37406 XUs and 71735 XVs -> 2067 XUVs
C:0 T:3 20810 XUs and 35308 XVs -> 152 XUVs
C:0 T:4 10762 XUs and 9284 XVs -> 91 XUVs
C:0 T:5 49001 XUs and 39725 XVs -> 6539 XUVs
C:0 T:6 120449 XUs and 148187 XVs -> 4839 XUVs
C:0 T:7 46479 XUs and 55922 XVs -> 461 XUVs
C:0 T:8 10633 XUs and 8871 XVs -> 107 XUVs
C:0 T:9 68952 XUs and 79443 XVs -> 6328 XUVs
C:0 T:10 52709 XUs and 35116 XVs -> 6233 XUVs
C:0 T:11 564 XUs and 462 XVs -> 54 XUVs
33078 XUVs total
11967 collection wire objects
30809 potential space points
Neighbour search...
2817537 tests to find 1472590 neighbours
Iterating with no regularization...
Begin: 6.57957e+07
0 5.45429e+07
1 5.38246e+07
2 5.37296e+07
3 5.3704e+07
Now with regularization...
Begin: 4.46697e+07
0 4.46281e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411421541
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411421541
fLifetime: 30000 [us]
30-Dec-2025 00:53:08 CET  Closed output file "reco_stage1.root"
30-Dec-2025 00:53:08 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          152.662       242.651       314.709       224.364       51.4053        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000856609   0.00317942    0.00485332    0.00327843    0.00107161       10     
reco:ophit:OpHitFinder                              0.1548       0.233837      0.312645      0.222763      0.0502675       10     
reco:opflash:OpFlashFinder                         0.0727355     0.127498      0.180813      0.134654      0.0306555       10     
reco:opslicer:OpSlicer                             0.331218      0.502442      0.789201      0.435716      0.168853        10     
reco:caldata:DataPrepModule                         3.37155       4.25957       5.56053       4.00404      0.802848        10     
reco:wclsdatasp:WireCellToolkit                     23.9639       28.6489       36.2498       25.6251       4.60464        10     
reco:gaushit:GausHitFinder                          0.49365      0.762282       1.05259      0.769709      0.131743        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000243748   0.000385443   0.000510339   0.000386932   7.09321e-05      10     
reco:reco3d:SpacePointSolver                        5.83593       10.6098       20.3436       8.26307       4.56507        10     
reco:hitpdune:DisambigFromSpacePoints              0.606901       1.20663       2.03748       1.11836      0.390879        10     
reco:pandora:StandardPandora                        8.13895       22.1779       35.4161       20.1075       7.95898        10     
reco:pandoraWriter:StandardPandora                 0.175652      0.317003       0.4469       0.312236      0.0763455       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.06374       2.44382       3.71649       2.22035      0.753173        10     
reco:pandoraShower:LArPandoraShowerCreation         1.06308       2.53665       3.87102       2.4713        0.8202         10     
reco:pandoracalo:Calorimetry                       0.669952       1.47105       2.23538       1.47303      0.420939        10     
reco:pandoracalonosce:Calorimetry                  0.605176       1.35828       2.05063       1.35085      0.395427        10     
reco:pandorapid:Chi2ParticleID                    0.00252693    0.00425826    0.00739258    0.00366023    0.00148802       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0223488     0.0390676     0.0591702     0.0372696     0.0107913       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00249689     0.0040427    0.00645654    0.00353147    0.00128153       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.14422       2.84792       4.45568       2.82604      0.940337        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.08383       2.74122       4.55135       2.81309      0.973673        10     
reco:emtrkmichelid:EmTrackMichelId                  85.5954       130.724       163.855       120.457       25.3481        10     
reco:crttag:SingleCRTMatchingProducer             0.00802087     0.012549      0.0184829     0.0130824    0.00377451       10     
reco:crtreco:TwoCRTMatchingProducer               0.00421975    0.00727281     0.0110954    0.00751351    0.00204339       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.44484      0.683322      0.805274      0.708871      0.106922        10     
reco:pandora2Track:LArPandoraTrackCreation          3.76906       8.3595        15.4542       7.76899       3.31869        10     
reco:pandora2calo:Calorimetry                       1.90482       4.35707       7.83993       4.38354       1.63709        10     
reco:pandora2calonosce:Calorimetry                  1.83027       3.93876       7.52816       3.87333       1.47116        10     
reco:pandora2pid:Chi2ParticleID                   0.00604795    0.00966053     0.0193174    0.00804075    0.00378485       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0299041     0.0489222     0.0897256     0.0413676     0.0169899       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00578454     0.0100585     0.0191253     0.0084453    0.00378001       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.49016       2.83775       5.4932        2.62178       1.11103        10     
reco:pandora2Showercalo:ShowerCalorimetry           1.67411       3.68244       6.62049       3.61437       1.38279        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.55977       3.40567       6.0809        3.32672       1.25245        10     
[art]:TriggerResults:TriggerResultInserter        3.5366e-05    5.10519e-05   9.8936e-05    4.38375e-05   2.0937e-05       10     
end_path:out1:RootOutput                           8.275e-06    1.18362e-05   2.6269e-05    9.7785e-06    5.10802e-06      10     
end_path:out1:RootOutput(write)                     1.75808       2.20451       3.48063       1.95607      0.521654        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4851.61 MB
  Peak resident set size usage (VmHWM): 4041.26 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.64 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9604460
-rw-r--r-- 1 neutrino002 neutrino 2276027630 Dec 30 00:11 detsim_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 2162506298 Dec 30 00:09 detsim_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 1948588107 Dec 30 00:53 reco_stage1.root
-rw-r--r-- 1 neutrino002 neutrino 1719613588 Dec 29 22:31 g4_stage2.root
-rw-r--r-- 1 neutrino002 neutrino 1655955089 Dec 29 22:21 g4_stage1.root
-rw-r--r-- 1 neutrino002 neutrino   71649400 Dec 30 00:50 Pandora_Events.pndr
-rw-r--r-- 1 neutrino002 neutrino     242745 Dec 30 00:53 jobscript.log
-rw-r--r-- 1 neutrino002 neutrino     217088 Dec 30 00:53 mem.db
-rw-r--r-- 1 neutrino002 neutrino      62376 Dec 30 00:53 Pandora_Geometry.xml
-rw-r--r-- 1 neutrino002 neutrino      36864 Dec 30 00:53 time.db
-rw-r--r-- 1 neutrino002 neutrino        519 Dec 30 00:11 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Dec 29 22:31 g4_protoDUNE_hist.root
-rw-r--r-- 1 neutrino002 neutrino        519 Dec 30 00:53 protoDUNE_reco1_hist.root
-rw-r--r-- 1 neutrino002 neutrino         52 Dec 29 22:14 all-input-dids.txt
-rw-r--r-- 1 neutrino002 neutrino          7 Dec 30 00:53 justin-processed-pfns.txt
-rw-r--r-- 1 neutrino002 neutrino          0 Dec 30 00:16 ProtoDUNE.log
-rw-r--r-- 1 neutrino002 neutrino          0 Dec 30 00:11 debugprod.log
justIN time: 2026-02-07 02:18:59 UTC       justIN version: 01.06.00