------------------------------------------------------ Job is running on node compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 ------------------------------------------------------ PBS: qsub is running on coma.hpc1.cs.cmu.edu PBS: originating queue is physics PBS: executing queue is physics PBS: working directory is /physics2/vat/bluetide/jobfiles PBS: execution mode is PBS_BATCH PBS: job identifier is 637149.coma.hpc1.cs.cmu.edu PBS: job name is job_dmo.sh PBS: node file is /var/spool/torque/aux//637149.coma.hpc1.cs.cmu.edu PBS: current home directory is /home/vat PBS: PATH = /home/vat/anaconda2/bin:/home/vat/anaconda2/bin:/opt/openmpi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/opt/dell/srvadmin/bin:/home/vat/bin:/home/vat/physics/vatcosmo/Mathematica/9.0/Executables/ ------------------------------------------------------ server is coma.hpc1.cs.cmu.edu workdir is /scratch/PBS_637149.coma.hpc1.cs.cmu.edu permdir is /home/vat/physics/vatcosmo servpermdir is coma.hpc1.cs.cmu.edu:/home/vat/physics/vatcosmo ------------------------------------------------------ Job is running on node compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-38 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-36 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-34 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-2-16 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-18 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-4 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-3 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 compute-1-7-2 ------------------------------------------------------ Transferring files from server to compute node PBS: working directory is /physics2/vat/bluetide/jobfiles Writing files in node directory /scratch/PBS_637149.coma.hpc1.cs.cmu.edu from /physics2/vat/bluetide/jobfiles and /home/vat/physics/vatcosmo /home/vat /physics2/vat/bluetide/jobfiles This is MP-Gadget, version 5.0.2a5febcd5d-dirty. Running on 128 MPI Ranks. 16 OpenMP Threads. Code was compiled with settings: mpiicc -openmp -O0 -g -DDENSITY_INDEPENDENT_SPH -DPETAPM_ORDER=1 -DOPENMP_USE_SPINLOCK -DUNEQUALSOFTENINGS -DSPH_GRAD_RHO -DTOPNODEFACTOR=5.0 -DMYSORT -DSFR -DMETALS -DWINDS -DBLACK_HOLES -DNO_ISEND_IRECV_IN_DOMAIN -DINHOMOG_GASDISTR_HINT Size of particle structure 208 [bytes] Size of blackhole structure 200 [bytes] Size of sph particle structure 216 [bytes] Triggering Action on `OutputList` Triggering Action on `StarformationCriterion` Triggering Action on `BlackHoleFeedbackMethod` [ 000000.00 ] Task 0: ----------- Running with Parameters ---------- InitCondFile /nfs/nas-0-1/vat/bluetide/ICDMO-400MPC/IC # Path to the Initial Condition File OutputDir /nfs/nas-0-1/vat/bluetide/RUNDMO-400MPC/ # Prefix to the output files TreeCoolFile # Path to the Cooling Table MetalCoolFile # Path to the Metal Cooling Table. Refer to cooling.c UVFluctuationFile # Path to the UVFluctation Table. Refer to cooling.c. DensityKernelType quintic # SnapshotFileBase snapshot # EnergyFile energy.txt # CpuFile cpu.txt # InfoFile info.txt # OutputList 0.01 # List of output times Omega0 0.2814 # CMBTemperature 2.7255 # Present-day CMB temperature in Kelvin, default from Fixsen 2009; affects background if RadiationOn is set. OmegaBaryon 0.0464 # OmegaLambda 0.7186 # HubbleParam 0.697 # BoxSize 400000 # MaxMemSizePerCore 1200 # CpuTimeBetRestartFile 9000 # TimeBegin 0.00625 # TimeMax 1 # TimeLimitCPU 86400 # DomainOverDecompositionFactor 8 # Number of sub domains on a MPI rank DomainReportSpeedfac 0 # Print speed factors in Domain, for profiling. TreeDomainUpdateFrequency 0.025 # ErrTolTheta 0.5 # TypeOfOpeningCriterion 1 # ErrTolIntAccuracy 0.02 # ErrTolForceAcc 0.005 # Nmesh 512 # MinGasHsmlFractional 0 # MaxGasVel 300000 # TypeOfTimestepCriterion 0 # Magic numbers! MaxSizeTimestep 0.1 # MinSizeTimestep 0 # MaxRMSDisplacementFac 0.2 # ArtBulkViscConst 0.75 # CourantFac 0.15 # DensityResolutionEta 1 # Resolution eta factor (See Price 2008) 1 = 33 for Cubic Spline DensityContrastLimit 100 # Max contrast for hydro force calculation MaxNumNgbDeviation 2 # HydroCostFactor 1 # Cost factor of hydro calculation, default to 1. NumPartPerFile 134217728 # number of particles per file NumWriters 128 # Number of concurrent writer processes. 0 implies Number of Tasks EnableAggregatedIO 1 # Use the Aggregated IO policy for small data set (Experimental). CoolingOn 0 # HydroOn 0 # StarformationOn 0 # RadiationOn 0 # Include radiation density in the background evolution. FastParticleType 2 # Particles of this type will not decrease the timestep. Default neutrinos. NoTreeType 2 # Particles of this type will not produce tree forces. Default neutrinos. SofteningHalo 1.85 # SofteningDisk 0 # SofteningBulge 0 # SofteningGas 1.85 # SofteningStars 1.85 # SofteningBndry 1.85 # SofteningHaloMaxPhys 1.85 # SofteningDiskMaxPhys 0 # SofteningBulgeMaxPhys 0 # SofteningGasMaxPhys 1.85 # SofteningStarsMaxPhys 1.85 # SofteningBndryMaxPhys 1.85 # BufferSize 100 # PartAllocFactor 2 # TopNodeAllocFactor 0.5 # InitGasTemp 580 # MinGasTemp 5 # RestartFromBlueTidesPhaseI 0 # The Restart snapshot is BlueTidesPhaseI, some fields are missing. SnapshotWithFOF 1 # Enable Friends-of-Friends halo finder. FOFHaloLinkingLength 0.2 # Linking length for Friends of Friends halos. FOFHaloMinLength 32 # MinFoFMassForNewSeed 5 # Minimal Mass for seeding tracer particles TimeBetweenSeedingSearch 1.05 # Time Between Seeding Attempts: default to a a large value, meaning never. BlackHoleOn 0 # Enable Blackhole BlackHoleAccretionFactor 100 # BlackHoleEddingtonFactor 3 # SeedBlackHoleMass 5e-05 # BlackHoleNgbFactor 2 # BlackHoleMaxAccretionRadius 99999 # BlackHoleFeedbackFactor 0.05 # BlackHoleFeedbackRadius 0 # BlackHoleFeedbackRadiusMaxPhys 0 # BlackHoleFeedbackMethod mass | spline # StarformationCriterion density | h2 # CritOverDensity 57.7 # CritPhysDensity 0 # FactorSN 0.1 # FactorEVP 1000 # TempSupernova 1e+08 # TempClouds 1000 # MaxSfrTimescale 1.5 # WindModel decouple | halo | ofjt10 | nowind # WindEfficiency 2 # WindEnergyFraction 1 # WindSigma0 353 # WindSpeedFactor 3.7 # WindFreeTravelLength 20 # WindFreeTravelDensFac 0.1 # QuickLymanAlphaProbability 0 # [ 000000.00 ] Task 0: ---------------------------------------------- [ 000000.00 ] Task 0: StarformationOn is disabled! [ 000000.00 ] Task 0: The Density Kernel type is QuinticSpline [ 000000.00 ] Task 0: The Density resolution is 1 * mean separation, or 113 neighbours [ 000000.00 ] Task 0: Reference Windspeed: 1306.1 [ 000000.00 ] Task 0: Hubble (internal units) = 0.1 [ 000000.00 ] Task 0: G (internal units) = 43007.1 [ 000000.00 ] Task 0: UnitMass_in_g = 1.989e+43 [ 000000.00 ] Task 0: UnitTime_in_s = 3.08568e+16 [ 000000.00 ] Task 0: UnitVelocity_in_cm_per_s = 100000 [ 000000.00 ] Task 0: UnitDensity_in_cgs = 6.76991e-22 [ 000000.00 ] Task 0: UnitEnergy_in_cgs = 1.989e+53 [ 000000.00 ] Task 0: Photon density OmegaG = 5.0888e-05 [ 000000.00 ] Task 0: Massless Neutrino density OmegaNu0 = 3.51282e-05 [ 000000.00 ] Task 0: Curvature density OmegaK = 0 [ 000000.00 ] Task 0: [ 000000.10 ] Aggregated IO is enabled [ 000000.26 ] Using 2D Task mesh 8 x 16 [ 000001.68 ] Reading snapshot /nfs/nas-0-1/vat/bluetide/ICDMO-400MPC/IC [ 000001.82 ] Total number of particles: 000000000085184000 [ 000001.84 ] Total number of gas particles: 000000000000000000 [ 000001.86 ] Total number of star particles: 000000000000000000 [ 000001.88 ] Total number of bh particles: 000000000000000000 [ 000002.41 ] Allocated 264.023 MByte for particle storage. [ 000002.41 ] Allocated 0 MByte for storage of SPH data. [ 000002.41 ] Allocated 0 MByte for storage of BH data. [ 000002.80 ] Reading Block 1/Position [ 000004.83 ] Reading Block 1/Velocity [ 000005.91 ] Reading Block 1/ID [ 000008.05 ] Testing ID uniqueness... [ 000009.98 ] success. took=1.93456 sec [ 000010.29 ] Blackholes: Eliminated 0 gas particles and merged away 0 black holes. [ 000010.79 ] domain decomposition... (presently allocated=279.445 MB) [ 000010.82 ] Allocated 33.0107 MByte for top-level domain structure [ 000010.89 ] use of 38.0852 MB of temporary storage for domain decomposition... (presently allocated=350.562 MB) At 'DOMAIN', domain_Decomposition()/domain.c/183: Largest Allocation = 350.566 Mbyte (on task=0), Smallest = 350.566 Mbyte, Average = 350.566 Mbyte ------------------------ Allocated Memory Blocks---------------------------------------- Task Nr F Variable MBytes Cumulative Function/File/Linenumber ---------------------------------------------------------------------------------------- 0 0 0 Powerspectrum 0.1250 0.1250 gravpm.c:30:powerspectrum_a 0 1 0 Powermodes 0.0625 0.1875 gravpm.c:32:powerspectrum_a 0 2 0 NextActiveParti 5.0781 5.2656 allocate.c:23:allocate_memory 0 3 0 NextInTimeBin 5.0781 10.3438 allocate.c:25:allocate_memory 0 4 0 PrevInTimeBin 5.0781 15.4219 allocate.c:27:allocate_memory 0 5 0 P 264.0234 279.4453 allocate.c:29:allocate_memory 0 6 0 SphP 0.0000 279.4453 allocate.c:50:allocate_memory 0 7 0 BhP 0.0000 279.4453 allocate.c:54:allocate_memory 0 8 0 DomainStartList 0.0039 279.4492 domain.c:258:domain_allocate 0 9 0 DomainEndList 0.0039 279.4531 domain.c:261:domain_allocate 0 10 0 TopNodes 33.0039 312.4570 domain.c:264:domain_allocate 0 11 0 list_cadj_cpu 0.0039 312.4609 domain.c:152:domain_Decompos 0 12 0 list_cadj_cost 0.0039 312.4648 domain.c:154:domain_Decompos 0 13 0 list_load 0.0039 312.4688 domain.c:156:domain_Decompos 0 14 0 list_loadsph 0.0039 312.4727 domain.c:158:domain_Decompos 0 15 0 list_work 0.0039 312.4766 domain.c:160:domain_Decompos 0 16 0 list_speedfac 0.0039 312.4805 domain.c:162:domain_Decompos 0 17 0 domainWork 2.5391 315.0195 domain.c:164:domain_Decompos 0 18 0 domainCount 2.5391 317.5586 domain.c:166:domain_Decompos 0 19 0 domainCountSph 2.5391 320.0977 domain.c:168:domain_Decompos 0 20 0 topNodes 30.4648 350.5625 domain.c:171:domain_Decompos ---------------------------------------------------------------------------------------- [ 000014.00 ] Before=4681 [ 000014.01 ] After=37449 [ 000014.02 ] NTopleaves= 32768 NTopnodes=37449 (space for 665501) [ 000014.92 ] work-load balance=1.00003 memory-balance=1.00003 [ 000015.72 ] iter=0 exchange of 0000084449211 particles [ 000019.43 ] domain decomposition done. (took 8.63075 sec) [ 000019.44 ] begin Peano-Hilbert order... [ 000021.07 ] Peano-Hilbert done. [ 000021.07 ] Freed 28.7499 MByte in top-level domain structure [ 000021.27 ] Allocated 172.966 MByte for BH-tree, and 0.125 Mbyte for top-leaves. (presently allocted 454.41 MB) [ 000021.27 ] Tree construction. (presently allocated=454.41 MB) [ 000021.33 ] Blackholes: Eliminated 0 gas particles and merged away 0 black holes. [ 000022.43 ] Tree construction done. At 'DENSITY', treewalk_run()/treewalk.c/503: Largest Allocation = 524.16 Mbyte (on task=18), Smallest = 524.156 Mbyte, Average = 524.16 Mbyte ------------------------ Allocated Memory Blocks---------------------------------------- Task Nr F Variable MBytes Cumulative Function/File/Linenumber ---------------------------------------------------------------------------------------- 18 0 0 Powerspectrum 0.1250 0.1250 gravpm.c:30:powerspectrum_a 18 1 0 Powermodes 0.0625 0.1875 gravpm.c:32:powerspectrum_a 18 2 0 NextActiveParti 5.0781 5.2656 allocate.c:23:allocate_memory 18 3 0 NextInTimeBin 5.0781 10.3438 allocate.c:25:allocate_memory 18 4 0 PrevInTimeBin 5.0781 15.4219 allocate.c:27:allocate_memory 18 5 0 P 264.0234 279.4453 allocate.c:29:allocate_memory 18 6 0 SphP 0.0000 279.4453 allocate.c:50:allocate_memory 18 7 0 BhP 0.0000 279.4453 allocate.c:54:allocate_memory 18 8 0 DomainStartList 0.0039 279.4492 domain.c:258:domain_allocate 18 9 0 DomainEndList 0.0039 279.4531 domain.c:261:domain_allocate 18 10 0 TopNodes 1.8594 281.3125 domain.c:264:domain_allocate 18 11 0 DomainNodeIndex 0.1250 281.4375 forcetree.c:1445:force_treealloc 18 12 0 Nodes_base 88.7305 370.1680 forcetree.c:1448:force_treealloc 18 13 0 Extnodes_base 73.9414 444.1094 forcetree.c:1455:force_treealloc 18 14 0 Nextnode 5.2227 449.3320 forcetree.c:1475:force_treealloc 18 15 0 Father 5.0781 454.4102 forcetree.c:1481:force_treealloc 18 16 0 Left 5.0781 459.4883 density.c.c:128:density 18 17 0 Right 5.0781 464.5664 density.c.c:129:density 18 18 0 Ngblist 40.6211 505.1875 treewalk.cc:139:ev_begin 18 19 0 DataIndexTable 4.4805 509.6680 treewalk.c:144:ev_begin 18 20 0 DataNodeList 11.9414 521.6094 treewalk.c:146:ev_begin 18 21 0 ActiveQueue 2.5391 524.1484 treewalk.c:270:treewalk_get_qu 18 22 0 PrimaryTasks 0.0000 524.1484 treewalk.c:153:ev_begin 18 23 0 currentIndexPer 0.0039 524.1523 treewalk.c:156:ev_begin 18 24 0 currentEndPerTh 0.0039 524.1562 treewalk.c:157:ev_begin 18 25 0 EvDataGet 0.0000 524.1562 treewalk.c:555:ev_get_remote ---------------------------------------------------------------------------------------- [ 000024.45 ] Converting u -> entropy, with density split sph [ 000026.82 ] iteration 000, max relative difference = 0 [ 000028.85 ] Setting next time for snapshot file to Time_next= 0.01 [ 000029.05 ] Begin Step 0, Time: 0.00625, Redshift: 159, Nf = 00000085184000, Systemstep: 0, Dloga: 0, status: PM-Step [ 000029.05 ] TotNumPart: 0000085184000 SPH 0000000000000 BH 0000000000 STAR 0000000000000 [ 000029.05 ] Occupied timebins: non-sph sph dt [ 000029.05 ] X bin= 0 00000085184000 00000000000000 0 [ 000029.05 ] ----------------------------------- [ 000029.05 ] Total:00000085184000 00000000000000 Sum:00000085184000 [ 000029.06 ] Start force computation... [ 000029.18 ] max number of regions is 260 At 'PetaPM', pm_alloc()/petapm.c/679: Largest Allocation = 320.508 Mbyte (on task=124), Smallest = 318.734 Mbyte, Average = 320.193 Mbyte ------------------------ Allocated Memory Blocks---------------------------------------- Task Nr F Variable MBytes Cumulative Function/File/Linenumber ---------------------------------------------------------------------------------------- 124 0 0 Powerspectrum 0.1250 0.1250 gravpm.c:30:powerspectrum_a 124 1 0 Powermodes 0.0625 0.1875 gravpm.c:32:powerspectrum_a 124 2 0 NextActiveParti 5.0781 5.2656 allocate.c:23:allocate_memory 124 3 0 NextInTimeBin 5.0781 10.3438 allocate.c:25:allocate_memory 124 4 0 PrevInTimeBin 5.0781 15.4219 allocate.c:27:allocate_memory 124 5 0 P 264.0234 279.4453 allocate.c:29:allocate_memory 124 6 0 SphP 0.0000 279.4453 allocate.c:50:allocate_memory 124 7 0 BhP 0.0000 279.4453 allocate.c:54:allocate_memory 124 8 0 DomainStartList 0.0039 279.4492 domain.c:258:domain_allocate 124 9 0 DomainEndList 0.0039 279.4531 domain.c:261:domain_allocate 124 10 0 TopNodes 1.8594 281.3125 domain.c:264:domain_allocate 124 11 0 PMreal 8.5000 289.8125 petapm.ce.c:667:pm_alloc 124 12 0 PMcomplex 8.5000 298.3125 petapm.ce.c:668:pm_alloc 124 13 0 PMrho_k 8.5000 306.8125 petapm.ce.c:669:pm_alloc 124 14 0 PMmesh 13.6914 320.5039 petapm.ce.c:678:pm_alloc ---------------------------------------------------------------------------------------- [ 000032.39 ] PetaPM: 0009362820/0011868192 Pencils and 0156480197 Cells ############ WARNING: EARLY TERMINATION ############# Transferring files from compute nodes to server Writing files in permanent directory /home/vat/physics/vatcosmo Final files in permanent data directory: ############ WARNING: EARLY TERMINATION ############# Transferring files from compute nodes to server Writing files in permanent directory /home/vat/physics/vatcosmo Final files in permanent data directory: APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6) Transferring files from compute nodes to server Writing files in permanent directory /home/vat/physics/vatcosmo Final files in permanent data directory: