Jobsub ID 236990.22@dunegpschedd02.fnal.gov

Jobsub ID	236990.22@dunegpschedd02.fnal.gov
Workflow ID	9538
Stage ID	1
User name	ykermaid@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	2
	GPU	No
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-11-03 22:09:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2025-11-04 01:47:12
From worker node	Hostname	lupin13.farm.particle.cz
	cpuinfo	AMD EPYC 7551 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	2
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-03 22:10:15
Input files	vd-protodune:np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230.hdf5
Jobscript	Exit code	0
	Real time	3h (12920s)
	CPU time	3h (12406s = 96%)
	Max RSS bytes	8583290880 (8185 MiB)
Outputting started	2025-11-04 01:45:36
Output files	vd-protodune-det-reco:np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230_reco_stage1_20251104T014524_keepup.root
Finished	2025-11-04 01:47:12
Saved logs	justin-logs:236990.22-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

c : 8.73 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.97 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.84 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.82 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.68 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.07 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.02 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.02 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.27 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.27 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.27 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.26 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[02:44:46.958] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[02:44:46.959] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[02:44:46.959] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[02:44:46.959] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[02:44:46.959] I [ timer  ] Timer: Total node execution : 62.279999639838934 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 37769 traces tagged "gauss"
FrameSaver: q=9.45106e+06 n=798763 tag=gauss
wclsFrameSaver: saving 46537 traces tagged "wiener"
FrameSaver: q=1.03254e+07 n=766964 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 337 XUs and 371 XVs -> 20 XUVs
C:0 T:1 60 XUs and 64 XVs -> 0 XUVs
C:0 T:2 566 XUs and 616 XVs -> 29 XUVs
C:0 T:3 287 XUs and 346 XVs -> 6 XUVs
C:0 T:4 552 XUs and 482 XVs -> 12 XUVs
C:0 T:5 66 XUs and 94 XVs -> 0 XUVs
C:0 T:6 2593 XUs and 2624 XVs -> 132 XUVs
C:0 T:7 11779 XUs and 10987 XVs -> 1212 XUVs
C:0 T:8 1989 XUs and 3761 XVs -> 127 XUVs
C:0 T:9 387 XUs and 626 XVs -> 22 XUVs
C:0 T:10 1252 XUs and 1844 XVs -> 63 XUVs
C:0 T:11 1021 XUs and 1581 XVs -> 97 XUVs
C:0 T:12 427 XUs and 560 XVs -> 26 XUVs
C:0 T:13 497 XUs and 1074 XVs -> 26 XUVs
C:0 T:14 1183 XUs and 1375 XVs -> 78 XUVs
C:0 T:15 365 XUs and 775 XVs -> 46 XUVs
1896 XUVs total
1017 collection wire objects
1896 potential space points
Neighbour search...
71916 tests to find 18766 neighbours
Iterating with no regularization...
Begin: 5.6181e+09
0 5.34726e+09
1 5.33641e+09
2 5.33582e+09
Now with regularization...
Begin: 5.20384e+09
0 5.20367e+09
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
04-Nov-2025 02:45:24 CET  Closed output file "np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230_reco_stage1_20251104T014524_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        99.1786       558.966       9572.55       132.171       1923.19        23     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.043e-05    6.28911e-05   0.000165169    5.602e-05    2.26592e-05      23     
produce:tpcrawdecoder:PDVDTPCReader               12.8426       15.0626       38.4625       13.7346       5.08246        23     
produce:triggerrawdecoder:PDVDTriggerReader4     0.460552      0.496441      0.569737      0.492669      0.024825        23     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000356298   0.000403999   0.000792867   0.000381038   8.76301e-05      23     
produce:ophit:OpHitFinder                        6.145e-05    9.31677e-05   0.000597237    6.615e-05    0.000107953      23     
produce:opflash:OpFlashFinderVerticalDrift      4.9369e-05    6.94982e-05   0.000395698    5.206e-05    6.98646e-05      23     
produce:wclsdatavd:WireCellToolkit                53.6264       75.8452       192.986       67.2305       27.9433        23     
produce:gaushit:GausHitFinder                    0.844788       1.35316       4.56395       1.14128       0.72473        23     
produce:nhitsfilter:NumberOfHitsFilter          0.000236139   0.000309672   0.000752767   0.000268049   0.000110681      23     
produce:reco3d:SpacePointSolver                   7.0049        16.1054       117.572       10.7515       21.9494        23     
produce:hitpdune:DisambigFromSpacePoints         0.0973224     0.284111       2.60027       0.15234      0.503547        23     
produce:pandora:StandardPandora                   14.9951       439.508       9199.81       25.5469       1868.81        23     
produce:pandoraTrack:LArPandoraTrackCreation     0.423982       2.17695       28.1403      0.901143       5.56227        23     
produce:pandoraGnocalo:GnocchiCalorimetry        0.0230887     0.0360318     0.117128      0.0308042     0.0183189       23     
[art]:TriggerResults:TriggerResultInserter       1.546e-05    2.05607e-05   6.0639e-05      1.8e-05     9.09553e-06      23     
end_path:out1:RootOutput                          3.2e-06      6.193e-06    1.7979e-05     6.15e-06     3.30863e-06      23     
end_path:out1:RootOutput(write)                   5.72653       8.0573        11.292        8.54395       1.4887         23     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10616.4 MB
  Peak resident set size usage (VmHWM): 8583.29 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230_reco_stage1_20251104T014524_keepup.root
\tHists: np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230_reco_stage1_20251104T014524_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230_reco_stage1_20251104T014524_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230_reco_stage1_20251104T014524_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762220726.0,
    "core.end_time": 1762220726.0,
    "core.events": [
      295832,
      295836,
      295840,
      295844,
      295848,
      295852,
      295856,
      295860,
      295864,
      295868,
      295872,
      295876,
      295880,
      295884,
      295888,
      295892,
      295896,
      295900,
      295904,
      295908,
      295912,
      295916,
      295920
    ],
    "core.event_count": 23,
    "core.first_event_number": 295832,
    "core.last_event_number": 295920,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230_reco_stage1_20251104T014524_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1762220726.0,
    "core.end_time": 1762220726.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40346
    ],
    "core.runs_subruns": [
      4034600001
    ],
    "dune.daq_test": false,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040346_3205_df-s04-d0_dw_0_20251103T085230.hdf5"
    }
  ]
}