Jobsub ID 36336.151@dunegpschedd02.fnal.gov
| Jobsub ID | 36336.151@dunegpschedd02.fnal.gov |
| Workflow ID | 2323 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:53:52 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce02 |
| Last heartbeat | 2025-09-08 15:52:28 |
| From worker node | Hostname | cn002.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:09:24 |
| Input files | vd-protodune:np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5363s) |
| CPU time | 1h (4980s = 92%) |
| Max RSS bytes | 3113234432 (2969 MiB) |
| Outputting started | 2025-09-08 15:38:48 |
| Output files | |
| Finished | 2025-09-08 15:52:28 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 10.28 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.62 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.46 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.96 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.94 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.77 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.68 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Aux::Resampler : 0.52 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Aux::Resampler : 0.5 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Aux::Resampler : 0.5 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Aux::Resampler : 0.49 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Gen::Retagger : 0.02 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:36:36.255] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:36:36.255] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:36:36.255] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:36:36.255] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:36:36.255] I [ timer ] Timer: Total node execution : 68.8199995495379 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 38671 traces tagged "gauss"
FrameSaver: q=6.76596e+06 n=833352 tag=gauss
wclsFrameSaver: saving 50278 traces tagged "wiener"
FrameSaver: q=7.32713e+06 n=790488 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1252 XUs and 866 XVs -> 41 XUVs
C:0 T:1 88 XUs and 81 XVs -> 2 XUVs
C:0 T:2 1607 XUs and 1464 XVs -> 108 XUVs
C:0 T:3 3810 XUs and 3471 XVs -> 366 XUVs
C:0 T:4 1300 XUs and 1681 XVs -> 83 XUVs
C:0 T:5 2482 XUs and 1552 XVs -> 127 XUVs
C:0 T:6 8610 XUs and 6572 XVs -> 659 XUVs
C:0 T:7 82 XUs and 101 XVs -> 4 XUVs
C:0 T:8 14440 XUs and 31565 XVs -> 3851 XUVs
C:0 T:9 1184 XUs and 1356 XVs -> 125 XUVs
C:0 T:10 249 XUs and 286 XVs -> 13 XUVs
C:0 T:11 2177 XUs and 3064 XVs -> 384 XUVs
C:0 T:12 2348 XUs and 4038 XVs -> 565 XUVs
C:0 T:13 4790 XUs and 5642 XVs -> 1217 XUVs
C:0 T:14 1038 XUs and 1473 XVs -> 100 XUVs
C:0 T:15 1623 XUs and 1284 XVs -> 272 XUVs
7917 XUVs total
2224 collection wire objects
7917 potential space points
Neighbour search...
654679 tests to find 293586 neighbours
Iterating with no regularization...
Begin: 7.20811e+08
0 6.32513e+08
1 6.23875e+08
2 6.2286e+08
3 6.22594e+08
Now with regularization...
Begin: 6.00976e+08
0 6.00781e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
08-Sep-2025 16:37:35 BST Closed output file "np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150_reco_stage1_20250908T153735_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 125.045 193.444 317.088 176.691 55.9681 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 6.9292e-05 0.000115445 0.000265096 0.000105394 3.5046e-05 27
produce:tpcrawdecoder:PDVDTPCReader 11.6826 13.3347 18.5852 12.7059 1.62056 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0329619 0.0353139 0.0453024 0.0333502 0.00425086 27
produce:pdvddaphne:DAPHNEReaderPDVD 4.46769 5.83175 7.19829 5.92055 0.66238 27
produce:ophit:OpHitFinder 0.0431658 0.0637365 0.0903161 0.0681873 0.0120117 27
produce:opflash:OpFlashFinderVerticalDrift 0.00893486 0.0173424 0.0264547 0.0183378 0.00484898 27
produce:wclsdatavd:WireCellToolkit 52.1856 72.2448 118.859 71.3791 12.1866 27
produce:gaushit:GausHitFinder 1.08818 1.63299 2.29347 1.64483 0.359895 27
produce:nhitsfilter:NumberOfHitsFilter 0.000381633 0.000577851 0.000845726 0.000583799 0.000116648 27
produce:reco3d:SpacePointSolver 7.42564 12.8583 16.9979 13.253 2.92622 27
produce:hitpdune:DisambigFromSpacePoints 0.145939 0.301186 0.483113 0.287175 0.10189 27
produce:pandora:StandardPandora 30.6245 79.9767 198.133 60.0889 47.4229 27
produce:pandoraTrack:LArPandoraTrackCreation 0.52751 2.35607 5.35741 2.26902 1.25279 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0188598 0.0396849 0.0609975 0.0403438 0.011707 27
[art]:TriggerResults:TriggerResultInserter 2.916e-05 5.38494e-05 0.000131831 5.191e-05 1.89961e-05 27
end_path:out1:RootOutput 3.94e-06 1.0622e-05 3.8491e-05 1.0036e-05 6.09705e-06 27
end_path:out1:RootOutput(write) 3.09305 4.6794 7.79078 5.02571 1.03356 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4998.92 MB
Peak resident set size usage (VmHWM): 3113.23 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150_reco_stage1_20250908T153735_keepup.root
\tHists: np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150_reco_stage1_20250908T153737_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150_reco_stage1_20250908T153735_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150_reco_stage1_20250908T153735_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757345858.0,
"core.end_time": 1757345858.0,
"core.events": [
883969,
883989,
884009,
884029,
884049,
884069,
884089,
884109,
884129,
884149,
884169,
884189,
884209,
884229,
884249,
884269,
884289,
884309,
884329,
884349,
884369,
884389,
884409,
884429,
884449,
884469,
884489
],
"core.event_count": 27,
"core.first_event_number": 883969,
"core.last_event_number": 884489,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150_reco_stage1_20250908T153737_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757345858.0,
"core.end_time": 1757345858.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_1687_df-s03-d3_dw_0_20250907T035150.hdf5"
}
]
} 