justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242280.1@dunegpschedd01.fnal.gov

Jobsub ID242280.1@dunegpschedd01.fnal.gov
Workflow ID9545
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-03 10:56:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-11-03 15:01:39
From worker nodeHostnamewn-da-07.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 10:58:35
Input filesmonte-carlo-009545-000001
JobscriptExit code0
Real time4h (14491s)
CPU time3h (14360s = 99%)
Max RSS bytes4542320640 (4331 MiB)
Outputting started2025-11-03 15:00:07
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09545/1/001/reco_stage1.root
Finished2025-11-03 15:01:39
Saved logsjustin-logs:242280.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 136598 traces, 0 summary
Retagger: tagging trace set: gauss with 103798 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 103798 traces tagged "gauss"
FrameSaver: q=3.67039e+06 n=1853313 tag=gauss
wclsFrameSaver: saving 136598 traces tagged "wiener"
FrameSaver: q=3.88638e+06 n=1643749 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33283 XUs and 34228 XVs -> 273 XUVs
C:0 T:1 61759 XUs and 61566 XVs -> 4180 XUVs
C:0 T:2 157812 XUs and 178051 XVs -> 7927 XUVs
C:0 T:3 70404 XUs and 68131 XVs -> 698 XUVs
C:0 T:4 8015 XUs and 8887 XVs -> 137 XUVs
C:0 T:5 34658 XUs and 37217 XVs -> 9012 XUVs
C:0 T:6 107420 XUs and 242006 XVs -> 15794 XUVs
C:0 T:7 35572 XUs and 57031 XVs -> 570 XUVs
C:0 T:8 7263 XUs and 6429 XVs -> 71 XUVs
C:0 T:9 50526 XUs and 54041 XVs -> 3878 XUVs
C:0 T:10 25160 XUs and 44597 XVs -> 6052 XUVs
C:0 T:11 24 XUs and 11 XVs -> 4 XUVs
48596 XUVs total
12136 collection wire objects
44788 potential space points
Neighbour search...
6215074 tests to find 2936624 neighbours
Iterating with no regularization...
Begin: 8.49427e+07
0 7.21702e+07
1 7.13621e+07
2 7.12359e+07
3 7.11981e+07
Now with regularization...
Begin: 6.07426e+07
0 6.0691e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410286426
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410286426
fLifetime: 30000 [us]
03-Nov-2025 16:00:05 CET  Closed output file "reco_stage1.root"
03-Nov-2025 16:00:05 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          403.115       474.216       688.538       441.237       79.8734        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00104575    0.00520999    0.00893077    0.00556247    0.00270873       10     
reco:ophit:OpHitFinder                             0.265976      0.301382      0.373488      0.301335      0.0276944       10     
reco:opflash:OpFlashFinder                         0.184279      0.226376      0.338996      0.219911      0.0422326       10     
reco:opslicer:OpSlicer                              0.64708      0.742932       1.0168       0.714894      0.099802        10     
reco:caldata:DataPrepModule                         4.49236       4.65974       4.84032       4.68725      0.125306        10     
reco:wclsdatasp:WireCellToolkit                     33.1185       56.0541       70.7514       61.2175       12.4666        10     
reco:gaushit:GausHitFinder                          1.21539       1.43262       1.90637       1.43226      0.196792        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000649644   0.000777714   0.00107729    0.000734807   0.000140541      10     
reco:reco3d:SpacePointSolver                        12.9241       19.6092       29.0851       18.8343       4.15433        10     
reco:hitpdune:DisambigFromSpacePoints               1.46417       1.90817       2.85456       1.78941      0.367515        10     
reco:pandora:StandardPandora                        34.8977       45.7588       74.223        40.3293       11.8667        10     
reco:pandoraWriter:StandardPandora                 0.401267      0.471743      0.645324      0.463697      0.0631807       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.85544       6.59629       12.4896       5.9907        2.12883        10     
reco:pandoraShower:LArPandoraShowerCreation         4.42539       7.96626       19.5851       6.50306       4.0149         10     
reco:pandoracalo:Calorimetry                        2.56976       3.43851       6.59712       3.09179       1.13223        10     
reco:pandoracalonosce:Calorimetry                   2.43443       3.19118       6.27762       2.82771       1.0838         10     
reco:pandorapid:Chi2ParticleID                    0.00556037    0.00679203    0.00854493    0.00665918    0.000724455      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0479779     0.0557367     0.0744653     0.0536264    0.00705392       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0054866    0.00640902     0.0073395    0.00643463    0.000489763      10     
reco:pandoraShowercalo:ShowerCalorimetry            4.54952       7.57434       17.8773       6.46428       3.53401        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.59018       7.46824       17.9288       6.42379       3.5792         10     
reco:emtrkmichelid:EmTrackMichelId                  205.207       234.898       303.498       229.116       26.5197        10     
reco:crttag:SingleCRTMatchingProducer             0.00987097     0.0153105     0.0235289     0.0147444     0.0045309       10     
reco:crtreco:TwoCRTMatchingProducer               0.00874469     0.0127686     0.0173774     0.0117696    0.00282118       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.780178      0.935748       1.22432      0.922412      0.120796        10     
reco:pandora2Track:LArPandoraTrackCreation          13.6276       22.6843       39.2057       20.9167       7.34464        10     
reco:pandora2calo:Calorimetry                       6.48509       10.3449       15.8756       9.15418       3.09573        10     
reco:pandora2calonosce:Calorimetry                  6.19269       9.82908       15.4451       8.57724       2.96036        10     
reco:pandora2pid:Chi2ParticleID                    0.013935      0.0168415     0.0218458     0.016311     0.00216739       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0620558     0.0736335     0.095743      0.0731537    0.00939075       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0139048     0.016887      0.0212517     0.0165123    0.00207411       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.85209       8.20574       14.8641       7.56416       2.84101        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.40084       8.61233       13.9483       7.95803       2.64435        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.93112       7.58059       12.6688       7.12675       2.34635        10     
[art]:TriggerResults:TriggerResultInserter        2.9926e-05    4.02056e-05   0.000106563   3.27475e-05   2.22149e-05      10     
end_path:out1:RootOutput                           5.781e-06    7.0198e-06    1.1603e-05    6.5475e-06    1.58093e-06      10     
end_path:out1:RootOutput(write)                     2.8752        3.34711       4.47279       3.34927      0.432759        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5345.26 MB
  Peak resident set size usage (VmHWM): 4542.32 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.92 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11534592
-rw-r--r--. 1 dune009 dune 2658105263 Nov  3 14:38 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2517944807 Nov  3 14:36 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2418217024 Nov  3 16:00 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2096904542 Nov  3 12:19 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2027790061 Nov  3 12:05 g4_stage1.root
-rw-r--r--. 1 dune009 dune   91848700 Nov  3 15:55 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242889 Nov  3 16:00 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  3 16:00 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  3 16:00 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  3 16:00 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  3 14:38 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  3 12:19 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  3 16:00 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  3 11:58 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  3 16:00 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  3 14:45 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  3 14:38 debugprod.log
justIN time: 2025-11-05 04:50:28 UTC       justIN version: 01.05.01