Jobsub ID 36348.6@dunegpschedd02.fnal.gov
| Jobsub ID | 36348.6@dunegpschedd02.fnal.gov |
| Workflow ID | 2330 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 12:17:56 |
| Site | UK_Lancaster |
| Entry | UBoone_UK_Lancaster_HEC_grendel_ce02 |
| Last heartbeat | 2025-09-08 15:52:40 |
| From worker node | Hostname | comp23-12 |
| cpuinfo | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:26:07 |
| Input files | vd-protodune:np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (4613s) |
| CPU time | 54m (3244s = 70%) |
| Max RSS bytes | 3106729984 (2962 MiB) |
| Outputting started | 2025-09-08 15:43:01 |
| Output files | |
| Finished | 2025-09-08 15:52:40 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ll=51 anode=0
[16:41:06.033] D [ glue ] <FrameFanin:nfsp> EOS at call=51 with 8
[16:41:06.033] D [ glue ] frame sink sees EOS
[16:41:06.033] D [ pgraph ] <Pgrapher:> graph execution complete
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.42 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9.03 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 9 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.95 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.83 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.65 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.19 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 7.98 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Aux::Resampler : 0.87 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Aux::Resampler : 0.83 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Aux::Resampler : 0.79 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Aux::Resampler : 0.78 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:41:06.033] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:41:06.033] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:41:06.033] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:41:06.033] I [ timer ] Timer: Total node execution : 73.35999874025583 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 36691 traces tagged "gauss"
FrameSaver: q=5.10708e+06 n=689473 tag=gauss
wclsFrameSaver: saving 43963 traces tagged "wiener"
FrameSaver: q=5.37544e+06 n=660115 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 865 XUs and 704 XVs -> 39 XUVs
C:0 T:1 365 XUs and 334 XVs -> 15 XUVs
C:0 T:2 1246 XUs and 1093 XVs -> 53 XUVs
C:0 T:3 1261 XUs and 847 XVs -> 39 XUVs
C:0 T:4 720 XUs and 748 XVs -> 25 XUVs
C:0 T:5 809 XUs and 937 XVs -> 47 XUVs
C:0 T:6 159 XUs and 135 XVs -> 10 XUVs
C:0 T:7 405 XUs and 440 XVs -> 36 XUVs
C:0 T:8 702 XUs and 1134 XVs -> 61 XUVs
C:0 T:9 694 XUs and 938 XVs -> 77 XUVs
C:0 T:10 1329 XUs and 1565 XVs -> 95 XUVs
C:0 T:11 623 XUs and 1326 XVs -> 38 XUVs
C:0 T:12 503 XUs and 638 XVs -> 43 XUVs
C:0 T:13 545 XUs and 458 XVs -> 49 XUVs
C:0 T:14 552 XUs and 637 XVs -> 57 XUVs
C:0 T:15 1754 XUs and 2599 XVs -> 183 XUVs
867 XUVs total
760 collection wire objects
867 potential space points
Neighbour search...
5059 tests to find 2176 neighbours
Iterating with no regularization...
Begin: 2.74806e+08
0 2.48916e+08
1 2.48456e+08
2 2.48454e+08
Now with regularization...
Begin: 2.44326e+08
0 2.44325e+08
08-Sep-2025 16:41:47 BST Closed output file "np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715_reco_stage1_20250908T154147_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 114.562 172.67 290.803 172.02 40.6435 26
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 7.9485e-05 0.000123385 0.000149588 0.000126123 1.40962e-05 26
produce:tpcrawdecoder:PDVDTPCReader 19.9234 50.4055 78.2513 50.7615 12.8708 26
produce:triggerrawdecoder:PDVDTriggerReader4 0.0494612 0.0564167 0.0753206 0.0507914 0.00900241 26
produce:pdvddaphne:DAPHNEReaderPDVD 4.66922 7.25852 11.9763 7.02514 1.64429 26
produce:ophit:OpHitFinder 0.0288307 0.0345573 0.0378056 0.0352716 0.00207184 26
produce:opflash:OpFlashFinderVerticalDrift 0.00463556 0.0109405 0.0142218 0.0111948 0.00203634 26
produce:wclsdatavd:WireCellToolkit 40.4359 51.148 73.9036 47.88 9.76161 26
produce:gaushit:GausHitFinder 0.714844 1.12634 1.69551 1.1287 0.269443 26
produce:nhitsfilter:NumberOfHitsFilter 0.000243071 0.000355234 0.000506345 0.000363033 7.30996e-05 26
produce:reco3d:SpacePointSolver 6.13202 10.5373 17.3986 10.9623 3.18951 26
produce:hitpdune:DisambigFromSpacePoints 0.102866 0.198351 0.351369 0.207208 0.0716479 26
produce:pandora:StandardPandora 15.565 47.6256 132.378 47.745 26.921 26
produce:pandoraTrack:LArPandoraTrackCreation 0.265537 0.803579 1.47715 0.796602 0.377949 26
produce:pandoraGnocalo:GnocchiCalorimetry 0.0129559 0.022105 0.0312673 0.0226982 0.00541694 26
[art]:TriggerResults:TriggerResultInserter 1.6711e-05 2.43058e-05 4.643e-05 2.3549e-05 5.84948e-06 26
end_path:out1:RootOutput 5.789e-06 8.31162e-06 1.9141e-05 7.5975e-06 3.12747e-06 26
end_path:out1:RootOutput(write) 2.88461 3.39848 4.98109 3.2583 0.56208 26
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5012.68 MB
Peak resident set size usage (VmHWM): 3106.73 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715_reco_stage1_20250908T154147_keepup.root
\tHists: np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715_reco_stage1_20250908T154148_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715_reco_stage1_20250908T154147_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715_reco_stage1_20250908T154147_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346108.0,
"core.end_time": 1757346108.0,
"core.events": [
1576365,
1576385,
1576405,
1576425,
1576445,
1576465,
1576485,
1576505,
1576525,
1576545,
1576565,
1576585,
1576605,
1576625,
1576645,
1576665,
1576685,
1576705,
1576725,
1576745,
1576765,
1576785,
1576805,
1576825,
1576845,
1576865
],
"core.event_count": 26,
"core.first_event_number": 1576365,
"core.last_event_number": 1576865,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715_reco_stage1_20250908T154148_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346108.0,
"core.end_time": 1757346108.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_3009_df-s02-d3_dw_0_20250908T084715.hdf5"
}
]
} 