justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14071.1@dunegpschedd02.fnal.gov

Jobsub ID14071.1@dunegpschedd02.fnal.gov
Workflow ID256
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-02 23:05:48
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-08-03 02:55:21
From worker nodeHostnamewn-la-25.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-02 23:06:39
Input filesmonte-carlo-000256-000001
JobscriptExit code0
Real time3h (13612s)
CPU time3h (13290s = 97%)
Max RSS bytes4190851072 (3996 MiB)
Outputting started2025-08-03 02:53:32
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00256/1/001/reco_stage1.root
Finished2025-08-03 02:55:21
Saved logsjustin-logs:14071.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 93123 traces, 0 summary
Retagger: tagging trace set: gauss with 72080 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72080 traces tagged "gauss"
FrameSaver: q=2.39742e+06 n=1258791 tag=gauss
wclsFrameSaver: saving 93123 traces tagged "wiener"
FrameSaver: q=2.5374e+06 n=1112373 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 16324 XUs and 11917 XVs -> 63 XUVs
C:0 T:1 27013 XUs and 20629 XVs -> 2173 XUVs
C:0 T:2 83255 XUs and 85349 XVs -> 3423 XUVs
C:0 T:3 43644 XUs and 38472 XVs -> 189 XUVs
C:0 T:4 7987 XUs and 5600 XVs -> 29 XUVs
C:0 T:5 28607 XUs and 15406 XVs -> 2624 XUVs
C:0 T:6 71271 XUs and 107188 XVs -> 5457 XUVs
C:0 T:7 24511 XUs and 33183 XVs -> 494 XUVs
C:0 T:8 5719 XUs and 5888 XVs -> 28 XUVs
C:0 T:9 32418 XUs and 39496 XVs -> 2673 XUVs
C:0 T:10 25448 XUs and 33982 XVs -> 5867 XUVs
C:0 T:11 610 XUs and 453 XVs -> 126 XUVs
23146 XUVs total
8686 collection wire objects
21110 potential space points
Neighbour search...
1605592 tests to find 827214 neighbours
Iterating with no regularization...
Begin: 5.5768e+07
0 4.66503e+07
1 4.58987e+07
2 4.57643e+07
3 4.57288e+07
Now with regularization...
Begin: 3.7963e+07
0 3.79178e+07
1 3.79007e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408425505
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408425505
fLifetime: 30000 [us]
03-Aug-2025 04:53:30 CEST  Closed output file "reco_stage1.root"
03-Aug-2025 04:53:30 CEST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          278.255       333.416       431.147       325.837       39.6803        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.0014582    0.00480133    0.00844615    0.00472159    0.00214546       10     
reco:ophit:OpHitFinder                             0.246253       0.36169      0.427062      0.382751      0.0622167       10     
reco:opflash:OpFlashFinder                         0.134016      0.230639      0.304533       0.25248      0.0557415       10     
reco:opslicer:OpSlicer                             0.435601      0.716389       0.90937       0.74853      0.148174        10     
reco:caldata:DataPrepModule                         4.84866       6.12658       6.86755       6.27534      0.523598        10     
reco:wclsdatasp:WireCellToolkit                      39.22        50.7958       60.148        51.0937       6.22479        10     
reco:gaushit:GausHitFinder                         0.846307       1.13001       1.30296       1.14185       0.1281         10     
reco:nhitsfilter:NumberOfHitsFilter               0.000341381   0.000583925   0.000831234   0.000616587   0.00015641       10     
reco:reco3d:SpacePointSolver                        8.85825       13.8643       21.7529       13.2722       4.06892        10     
reco:hitpdune:DisambigFromSpacePoints              0.830122       1.17717        1.58         1.21928      0.236243        10     
reco:pandora:StandardPandora                        18.364        25.2428       36.5539       23.9809       5.22021        10     
reco:pandoraWriter:StandardPandora                 0.353402      0.418293      0.495458      0.416632      0.0379992       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.08566        4.996        7.07704       4.64358       1.37884        10     
reco:pandoraShower:LArPandoraShowerCreation         3.32444       5.57733       8.41798       5.05279       1.76104        10     
reco:pandoracalo:Calorimetry                        1.62643       2.54702       3.38646       2.59433      0.588016        10     
reco:pandoracalonosce:Calorimetry                   1.51178       2.36672       3.23411       2.40146      0.545403        10     
reco:pandorapid:Chi2ParticleID                    0.00406018    0.00562331    0.00936694    0.00532617    0.00142114       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0389239     0.0466443     0.0535683     0.0462327    0.00401446       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00453726    0.00543034    0.00606542    0.00544743    0.000511129      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.86329       4.77082       7.42024       4.52034       1.4011         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.62617       4.77841       6.5529        4.76576       1.28418        10     
reco:emtrkmichelid:EmTrackMichelId                  134.079       159.899       201.926       161.696       20.239         10     
reco:crttag:SingleCRTMatchingProducer              0.0085936     0.014214      0.0198859     0.0134204    0.00423943       10     
reco:crtreco:TwoCRTMatchingProducer               0.00810491     0.0130672     0.0261273     0.0117289    0.00499127       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.666672      0.841637       1.02852      0.796194      0.120962        10     
reco:pandora2Track:LArPandoraTrackCreation          7.86216       14.4604       26.7594       14.218        5.15746        10     
reco:pandora2calo:Calorimetry                       3.82767       6.67927       11.6422       6.12569       2.30327        10     
reco:pandora2calonosce:Calorimetry                  3.84558       6.63184       12.1977       6.4115        2.29808        10     
reco:pandora2pid:Chi2ParticleID                    0.0087012     0.0113057     0.0134377     0.0112784    0.00164588       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0459187     0.0564319     0.0685626     0.0558663    0.00752234       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00895381     0.0113697     0.0134753     0.0115227    0.00154952       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.44732       5.38522       10.2339       4.94222       1.87747        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.83529       5.93817       9.59836       5.68411       1.79995        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.22582       5.32727       9.4187        5.08828       1.86001        10     
[art]:TriggerResults:TriggerResultInserter        3.8398e-05    5.94168e-05   0.00014123    5.0095e-05    2.86199e-05      10     
end_path:out1:RootOutput                           7.551e-06    9.8379e-06    1.6935e-05    8.5935e-06    3.02639e-06      10     
end_path:out1:RootOutput(write)                     2.21186       2.76954       3.42444       2.83313      0.388134        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4990.19 MB
  Peak resident set size usage (VmHWM): 4190.85 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.79 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9539912
-rw-r--r--. 1 dune003 dune 2256425182 Aug  3 03:55 detsim_stage2.root
-rw-r--r--. 1 dune003 dune 2143900587 Aug  3 03:53 detsim_stage1.root
-rw-r--r--. 1 dune003 dune 1897793226 Aug  3 04:53 reco_stage1.root
-rw-r--r--. 1 dune003 dune 1742575843 Aug  3 01:29 g4_stage2.root
-rw-r--r--. 1 dune003 dune 1664008387 Aug  3 01:13 g4_stage1.root
-rw-r--r--. 1 dune003 dune   63545700 Aug  3 04:50 Pandora_Events.pndr
-rw-r--r--. 1 dune003 dune     243192 Aug  3 04:53 jobscript.log
-rw-r--r--. 1 dune003 dune     217088 Aug  3 04:53 mem.db
-rw-r--r--. 1 dune003 dune      62376 Aug  3 04:53 Pandora_Geometry.xml
-rw-r--r--. 1 dune003 dune      36864 Aug  3 04:53 time.db
-rw-r--r--. 1 dune003 dune        519 Aug  3 03:55 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  3 01:29 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  3 04:53 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune003 dune         52 Aug  3 01:06 all-input-dids.txt
-rw-r--r--. 1 dune003 dune          7 Aug  3 04:53 justin-processed-pfns.txt
-rw-r--r--. 1 dune003 dune          0 Aug  3 04:01 ProtoDUNE.log
-rw-r--r--. 1 dune003 dune          0 Aug  3 03:55 debugprod.log
justIN time: 2025-08-04 16:02:45 UTC       justIN version: 01.04.00