justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.36@dunegpschedd02.fnal.gov

Jobsub ID263305.36@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-18 22:01:19
From worker nodeHostnamedunegli-8108645-0-fnpc22001.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 21:01:54
Input filesusertests:pdhd_prod_beam__261612_6_1_20251212T155435Z_gen_g4_IonScintPDExt.root_262727_61_1_20251215T214708Z_PDInt.root
JobscriptExit code0
Real time58m (3510s)
CPU time57m (3444s = 98%)
Max RSS bytes7562403840 (7212 MiB)
Outputting started2025-12-18 22:00:25
Output filesusertests:pdhd_prod_beam__261612_6_1_20251212T155435Z_gen_g4_IonScintPDExt.root_262727_61_1_20251215T214708Z_PDInt.root_263305_36_1_20251218T210157Z_detsim_reco1.root
Finished2025-12-18 22:01:19
Saved logsjustin-logs:263305.36-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 47197 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":22313 [0] "wiener0":24884 [24884] ] cmm:[ bad:0 lf_noisy:260 ] input 1: frame: ident=179 time=0 tick=512 with 30004 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13107 [0] "wiener1":16897 [16897] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 52970 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":22951 [0] "wiener2":30019 [30019] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 37938 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16459 [0] "wiener3":21479 [21479] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 168109 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":22313 [0] "gauss1":13107 [0] "gauss2":22951 [0] "gauss3":16459 [0] "wiener0":24884 [24884] "wiener1":16897 [16897] "wiener2":30019 [30019] "wiener3":21479 [21479] ] cmm:[ bad:0 lf_noisy:260 ] 
Retagger: tagging trace set: wiener with 93279 traces, 93279 summary
Retagger: tagging trace set: gauss with 74830 traces, 0 summary
[22:00:01.092] D [  glue  ] sink frame: #179 @0 with 168109 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[22:00:01.092] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[22:00:01.092] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[22:00:01.092] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[22:00:01.092] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[22:00:01.092] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[22:00:01.092] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[22:00:01.092] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[22:00:01.092] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[22:00:01.092] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[22:00:01.092] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[22:00:01.092] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[22:00:01.092] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[22:00:01.092] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[22:00:01.092] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[22:00:01.092] D [  glue  ] frame sink sees EOS
[22:00:01.092] D [ pgraph ] <Pgrapher:> graph execution complete 
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.56 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.54 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.31 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.18 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.96 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.94 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.91 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.9 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:00:01.092] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[22:00:01.092] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[22:00:01.092] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[22:00:01.092] I [ timer  ] Timer: Total node execution : 32.32000019028783 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74830 traces tagged "gauss"
FrameSaver: q=2.27113e+06 n=1173939 tag=gauss
wclsFrameSaver: saving 93279 traces tagged "wiener"
FrameSaver: q=2.80368e+06 n=1132332 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 14109 XUs and 7908 XVs -> 264 XUVs
C:0 T:3 23 XUs and 22 XVs -> 0 XUVs
C:0 T:4 7 XUs and 7 XVs -> 0 XUVs
C:0 T:5 54846 XUs and 45313 XVs -> 975 XUVs
C:0 T:6 18158 XUs and 18552 XVs -> 1514 XUVs
C:0 T:7 17 XUs and 12 XVs -> 0 XUVs
2753 XUVs total
1340 collection wire objects
2753 potential space points
Neighbour search...
59451 tests to find 35710 neighbours
Iterating with no regularization...
Begin: 3.13672e+07
0 3.0442e+07
1 3.03698e+07
2 3.03604e+07
Now with regularization...
Begin: 2.84803e+07
0 2.8476e+07
18-Dec-2025 22:00:24 UTC  Closed output file "pdhd_prod_beam__261612_6_1_20251212T155435Z_gen_g4_IonScintPDExt.root_262727_61_1_20251215T214708Z_PDInt.root_263305_36_1_20251218T210157Z_detsim_reco1.root"
18-Dec-2025 22:00:24 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           46.4157       56.0191       67.9216       56.3132       5.58851        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000360114    0.0024676    0.00398755    0.00275666    0.00116183       10     
reco:wclsdatahd:WireCellToolkit                      32.9491       33.4125       34.5507       33.2826      0.467893        10     
reco:wirefilter:WireFilter                         0.00742408     0.010071      0.0122308     0.010653     0.00172302       10     
reco:gaushit:GausHitFinder                          0.253851      0.358291      0.490515      0.356653      0.0710009       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000140738   0.000199617   0.00034219    0.000165505   6.96064e-05      10     
reco:reco3d:SpacePointSolver                        0.0897307     0.335751       0.92713      0.259298       0.22839        10     
reco:hitpdune:DisambigFromSpacePoints               0.103161      0.251213      0.449655      0.247822      0.0989772       10     
reco:pandora:StandardPandora                         1.98296       4.36654       7.57151       4.14013       1.46478        10     
reco:pandoraWriter:StandardPandora                  0.0625276     0.105832      0.150658       0.10059      0.0233225       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.284446      0.438303      0.603591      0.439193      0.094799        10     
reco:pandoraShower:LArPandoraShowerCreation         0.215454      0.457725      0.975923      0.459497      0.213056        10     
reco:pandoracalo:Calorimetry                         0.19471      0.278585      0.383023      0.280035      0.0557945       10     
reco:pandoracalonosce:Calorimetry                   0.179855      0.268771      0.372942      0.266046      0.0572647       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.205146      0.480127       1.05372      0.473927      0.229806        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.204632      0.475918       1.03963      0.464858      0.227552        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0183345     0.0264296     0.0389866     0.0253955    0.00540829       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0173782     0.0256622     0.0406696     0.0236724     0.006155        10     
reco:emtrkmichelid:EmTrackMichelId                   9.08602       13.9981       18.8632       14.2649       2.61962        10     
[art]:TriggerResults:TriggerResultInserter         1.3716e-05    2.03358e-05   5.4243e-05    1.64915e-05   1.14986e-05      10     
end_path:out1:RootOutput                            2.815e-06    4.8291e-06    2.0109e-05     3.021e-06    5.10168e-06      10     
end_path:out1:RootOutput(write)                     0.508862      0.697793      0.968202      0.698093       0.12061        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6318.16 MB
  Peak resident set size usage (VmHWM): 4150.47 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12229336
-rw-r--r-- 1 dunegli fnalgrid     231076 Dec 18 22:00 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        199 Dec 18 22:00 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:00 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 22:00 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 22:00 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 22:00 time.db
-rw-r--r-- 1 dunegli fnalgrid 6336989415 Dec 18 22:00 pdhd_prod_beam__261612_6_1_20251212T155435Z_gen_g4_IonScintPDExt.root_262727_61_1_20251215T214708Z_PDInt.root_263305_36_1_20251218T210157Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   34555090 Dec 18 22:00 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 21:49 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:49 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6150833572 Dec 18 21:49 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Dec 18 21:01 all-input-dids.txt
justIN time: 2025-12-18 23:05:56 UTC       justIN version: 01.05.03