------------------------------------------------------ Job is running on node compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 ------------------------------------------------------ PBS: qsub is running on coma.hpc1.cs.cmu.edu PBS: originating queue is bigmem PBS: executing queue is bigmem PBS: working directory is /physics2/vat/bluetide/jobfiles PBS: execution mode is PBS_BATCH PBS: job identifier is 636576.coma.hpc1.cs.cmu.edu PBS: job name is job_sim.sh PBS: node file is /var/spool/torque/aux//636576.coma.hpc1.cs.cmu.edu PBS: current home directory is /home/vat PBS: PATH = /home/yfeng1/local/bin:/opt/gcc/4.7.3/bin:/opt/intel/impi/5.0.1.035/intel64/bin:/opt/intel/Compiler/11.1/064/bin/intel64:/opt/intel/Compiler/11.1/064/bin/intel64:/home/vat/anaconda2/bin:/home/vat/anaconda2/bin:/opt/openmpi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbin:/opt/dell/srvadmin/bin:/home/vat/bin:/home/vat/physics/vatcosmo/Mathematica/9.0/Executables/ ------------------------------------------------------ server is coma.hpc1.cs.cmu.edu workdir is /scratch/PBS_636576.coma.hpc1.cs.cmu.edu permdir is /home/vat/physics/vatcosmo servpermdir is coma.hpc1.cs.cmu.edu:/home/vat/physics/vatcosmo ------------------------------------------------------ Job is running on node compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 compute-2-1 ------------------------------------------------------ Transferring files from server to compute node PBS: working directory is /physics2/vat/bluetide/jobfiles Writing files in node directory /scratch/PBS_636576.coma.hpc1.cs.cmu.edu from /physics2/vat/bluetide/jobfiles and /home/vat/physics/vatcosmo /home/vat /physics2/vat/bluetide/jobfiles This is MP-Gadget, version 5.0.2a5febcd5d-dirty. Running on 32 MPI Ranks. 32 OpenMP Threads. Code was compiled with settings: mpiicc -openmp -O0 -g -DDENSITY_INDEPENDENT_SPH -DPETAPM_ORDER=1 -DOPENMP_USE_SPINLOCK -DUNEQUALSOFTENINGS -DSPH_GRAD_RHO -DTOPNODEFACTOR=5.0 -DMYSORT -DSFR -DMETALS -DWINDS -DBLACK_HOLES -DNO_ISEND_IRECV_IN_DOMAIN -DINHOMOG_GASDISTR_HINT Size of particle structure 208 [bytes] Size of blackhole structure 200 [bytes] Size of sph particle structure 216 [bytes] Triggering Action on `OutputList` Triggering Action on `StarformationCriterion` Triggering Action on `BlackHoleFeedbackMethod` [ 000000.00 ] Task 0: ----------- Running with Parameters ---------- InitCondFile /nfs/nas-0-1/vat/bluetide/IC-10MPC/IC # Path to the Initial Condition File OutputDir /nfs/nas-0-1/vat/bluetide/RUN-10MPC/ # Prefix to the output files TreeCoolFile /physics2/vat/bluetide/inputfiles/TREECOOL_fg_june11 # Path to the Cooling Table MetalCoolFile /physics2/vat/bluetide/inputfiles/cooling_metal_UVB.hdf5 # Path to the Metal Cooling Table. Refer to cooling.c UVFluctuationFile # Path to the UVFluctation Table. Refer to cooling.c. DensityKernelType quintic # SnapshotFileBase snapshot # EnergyFile energy.txt # CpuFile cpu.txt # InfoFile info.txt # OutputList 0.1 # List of output times Omega0 0.2814 # CMBTemperature 2.7255 # Present-day CMB temperature in Kelvin, default from Fixsen 2009; affects background if RadiationOn is set. OmegaBaryon 0.0464 # OmegaLambda 0.7186 # HubbleParam 0.697 # BoxSize 10000 # MaxMemSizePerCore 1200 # CpuTimeBetRestartFile 9000 # TimeBegin 0.00625 # TimeMax 1 # TimeLimitCPU 43000 # DomainOverDecompositionFactor 8 # Number of sub domains on a MPI rank DomainReportSpeedfac 0 # Print speed factors in Domain, for profiling. TreeDomainUpdateFrequency 0.025 # ErrTolTheta 0.5 # TypeOfOpeningCriterion 1 # ErrTolIntAccuracy 0.02 # ErrTolForceAcc 0.005 # Nmesh 256 # MinGasHsmlFractional 0 # MaxGasVel 300000 # TypeOfTimestepCriterion 0 # Magic numbers! MaxSizeTimestep 0.1 # MinSizeTimestep 0 # MaxRMSDisplacementFac 0.2 # ArtBulkViscConst 0.75 # CourantFac 0.15 # DensityResolutionEta 1 # Resolution eta factor (See Price 2008) 1 = 33 for Cubic Spline DensityContrastLimit 100 # Max contrast for hydro force calculation MaxNumNgbDeviation 2 # HydroCostFactor 1 # Cost factor of hydro calculation, default to 1. NumPartPerFile 134217728 # number of particles per file NumWriters 32 # Number of concurrent writer processes. 0 implies Number of Tasks EnableAggregatedIO 1 # Use the Aggregated IO policy for small data set (Experimental). CoolingOn 1 # HydroOn 1 # StarformationOn 1 # RadiationOn 0 # Include radiation density in the background evolution. FastParticleType 2 # Particles of this type will not decrease the timestep. Default neutrinos. NoTreeType 2 # Particles of this type will not produce tree forces. Default neutrinos. SofteningHalo 1.85 # SofteningDisk 0 # SofteningBulge 0 # SofteningGas 1.85 # SofteningStars 1.85 # SofteningBndry 1.85 # SofteningHaloMaxPhys 1.85 # SofteningDiskMaxPhys 0 # SofteningBulgeMaxPhys 0 # SofteningGasMaxPhys 1.85 # SofteningStarsMaxPhys 1.85 # SofteningBndryMaxPhys 1.85 # BufferSize 100 # PartAllocFactor 2 # TopNodeAllocFactor 0.5 # InitGasTemp 580 # MinGasTemp 5 # RestartFromBlueTidesPhaseI 0 # The Restart snapshot is BlueTidesPhaseI, some fields are missing. SnapshotWithFOF 1 # Enable Friends-of-Friends halo finder. FOFHaloLinkingLength 0.2 # Linking length for Friends of Friends halos. FOFHaloMinLength 32 # MinFoFMassForNewSeed 5 # Minimal Mass for seeding tracer particles TimeBetweenSeedingSearch 1.05 # Time Between Seeding Attempts: default to a a large value, meaning never. BlackHoleOn 1 # Enable Blackhole BlackHoleAccretionFactor 100 # BlackHoleEddingtonFactor 3 # SeedBlackHoleMass 5e-05 # BlackHoleNgbFactor 2 # BlackHoleMaxAccretionRadius 99999 # BlackHoleFeedbackFactor 0.05 # BlackHoleFeedbackRadius 0 # BlackHoleFeedbackRadiusMaxPhys 0 # BlackHoleFeedbackMethod mass | spline # StarformationCriterion density | h2 # CritOverDensity 57.7 # CritPhysDensity 0 # FactorSN 0.1 # FactorEVP 1000 # TempSupernova 1e+08 # TempClouds 1000 # MaxSfrTimescale 1.5 # WindModel decouple | halo | ofjt10 | nowind # WindEfficiency 2 # WindEnergyFraction 1 # WindSigma0 353 # WindSpeedFactor 3.7 # WindFreeTravelLength 20 # WindFreeTravelDensFac 0.1 # QuickLymanAlphaProbability 0 # [ 000000.00 ] Task 0: ---------------------------------------------- [ 000000.00 ] Task 0: The Density Kernel type is QuinticSpline [ 000000.00 ] Task 0: The Density resolution is 1 * mean separation, or 113 neighbours [ 000000.00 ] Task 0: Reference Windspeed: 1306.1 [ 000000.00 ] Task 0: Hubble (internal units) = 0.1 [ 000000.00 ] Task 0: G (internal units) = 43007.1 [ 000000.00 ] Task 0: UnitMass_in_g = 1.989e+43 [ 000000.00 ] Task 0: UnitTime_in_s = 3.08568e+16 [ 000000.00 ] Task 0: UnitVelocity_in_cm_per_s = 100000 [ 000000.00 ] Task 0: UnitDensity_in_cgs = 6.76991e-22 [ 000000.00 ] Task 0: UnitEnergy_in_cgs = 1.989e+53 [ 000000.00 ] Task 0: Photon density OmegaG = 5.0888e-05 [ 000000.00 ] Task 0: Massless Neutrino density OmegaNu0 = 3.51282e-05 [ 000000.00 ] Task 0: Curvature density OmegaK = 0 [ 000000.00 ] Task 0: [ 000000.01 ] Aggregated IO is enabled [ 000000.03 ] Read ionization table with 236 entries in file `/physics2/vat/bluetide/inputfiles/TREECOOL_fg_june11'. [ 000000.06 ] Using UNIFORM UV BG from /physics2/vat/bluetide/inputfiles/TREECOOL_fg_june11 [ 000000.06 ] A0= 1000 [ 000000.06 ] Computed: PhysDensThresh= 0.000858616 (int units) 0.264121 h^2 cm^-3 [ 000000.06 ] EXPECTED FRACTION OF COLD GAS AT THRESHOLD = 0.904362 [ 000000.06 ] tcool=4.69337e-06 dens=0.027755 egyhot=2104.92 [ 000000.07 ] Run-away sets in for dens=0.414425 [ 000000.07 ] Dynamic range for quiescent star formation= 482.666 [ 000000.07 ] Isotherm sheet central density: 0.0481412 z0=0.0519306 [ 000000.30 ] Using 2D Task mesh 4 x 8 [ 000000.42 ] Reading snapshot /nfs/nas-0-1/vat/bluetide/IC-10MPC/IC [ 000000.49 ] Total number of particles: 000000000004194304 [ 000000.51 ] Total number of gas particles: 000000000002097152 [ 000000.51 ] Total number of star particles: 000000000000000000 [ 000000.51 ] Total number of bh particles: 000000000000000000 [ 000000.64 ] Allocated 52 MByte for particle storage. [ 000000.64 ] Allocated 54 MByte for storage of SPH data. [ 000000.64 ] Allocated 4.99992 MByte for storage of BH data. [ 000001.01 ] Reading Block 0/Position [ 000001.20 ] Reading Block 0/Velocity [ 000001.33 ] Reading Block 0/ID [ 000001.49 ] Reading Block 1/Position [ 000001.62 ] Reading Block 1/Velocity [ 000001.76 ] Reading Block 1/ID [ 000003.50 ] Testing ID uniqueness... [ 000003.60 ] success. took=0.101287 sec [ 000004.01 ] Blackholes: Eliminated 0 gas particles and merged away 0 black holes. [ 000004.52 ] domain decomposition... (presently allocated=114.344 MB) [ 000004.54 ] Allocated 6.502 MByte for top-level domain structure [ 000004.56 ] use of 7.50128 MB of temporary storage for domain decomposition... (presently allocated=128.395 MB) At 'DOMAIN', domain_Decomposition()/domain.c/183: Largest Allocation = 128.398 Mbyte (on task=0), Smallest = 128.398 Mbyte, Average = 128.398 Mbyte ------------------------ Allocated Memory Blocks---------------------------------------- Task Nr F Variable MBytes Cumulative Function/File/Linenumber ---------------------------------------------------------------------------------------- 0 0 0 BetaH0 0.0156 0.0156 cooling.c:568:InitCoolMemory 0 1 0 BetaHep 0.0156 0.0312 cooling.c:569:InitCoolMemory 0 2 0 AlphaHp 0.0156 0.0469 cooling.c:570:InitCoolMemory 0 3 0 AlphaHep 0.0156 0.0625 cooling.c:571:InitCoolMemory 0 4 0 Alphad 0.0156 0.0781 cooling.c:572:InitCoolMemory 0 5 0 AlphaHepp 0.0156 0.0938 cooling.c:573:InitCoolMemory 0 6 0 GammaeH0 0.0156 0.1094 cooling.c:574:InitCoolMemory 0 7 0 GammaeHe0 0.0156 0.1250 cooling.c:575:InitCoolMemory 0 8 0 GammaeHep 0.0156 0.1406 cooling.c:576:InitCoolMemory 0 9 0 Betaff 0.0156 0.1562 cooling.c:577:InitCoolMemory 0 10 0 Powerspectrum 0.1250 0.2812 gravpm.c:30:powerspectrum_a 0 11 0 Powermodes 0.0625 0.3438 gravpm.c:32:powerspectrum_a 0 12 0 NextActiveParti 1.0000 1.3438 allocate.c:23:allocate_memory 0 13 0 NextInTimeBin 1.0000 2.3438 allocate.c:25:allocate_memory 0 14 0 PrevInTimeBin 1.0000 3.3438 allocate.c:27:allocate_memory 0 15 0 P 52.0000 55.3438 allocate.c:29:allocate_memory 0 16 0 SphP 54.0000 109.3438 allocate.c:50:allocate_memory 0 17 0 BhP 5.0000 114.3438 allocate.c:54:allocate_memory 0 18 0 DomainStartList 0.0039 114.3477 domain.c:258:domain_allocate 0 19 0 DomainEndList 0.0039 114.3516 domain.c:261:domain_allocate 0 20 0 TopNodes 6.5039 120.8555 domain.c:264:domain_allocate 0 21 0 list_cadj_cpu 0.0039 120.8594 domain.c:152:domain_Decompos 0 22 0 list_cadj_cost 0.0039 120.8633 domain.c:154:domain_Decompos 0 23 0 list_load 0.0039 120.8672 domain.c:156:domain_Decompos 0 24 0 list_loadsph 0.0039 120.8711 domain.c:158:domain_Decompos 0 25 0 list_work 0.0039 120.8750 domain.c:160:domain_Decompos 0 26 0 list_speedfac 0.0039 120.8789 domain.c:162:domain_Decompos 0 27 0 domainWork 0.5039 121.3828 domain.c:164:domain_Decompos 0 28 0 domainCount 0.5039 121.8867 domain.c:166:domain_Decompos 0 29 0 domainCountSph 0.5039 122.3906 domain.c:168:domain_Decompos 0 30 0 topNodes 6.0039 128.3945 domain.c:171:domain_Decompos ---------------------------------------------------------------------------------------- [ 000007.61 ] Before=585 [ 000007.61 ] After=4681 [ 000007.61 ] NTopleaves= 4096 NTopnodes=4681 (space for 131073) [ 000008.38 ] work-load balance=1.00024 memory-balance=1.00024 [ 000008.82 ] iter=0 exchange of 0000004037162 particles [ 000009.62 ] domain decomposition done. (took 5.08852 sec) [ 000009.63 ] begin Peano-Hilbert order... [ 000010.92 ] Peano-Hilbert done. [ 000010.92 ] Freed 5.78577 MByte in top-level domain structure [ 000010.95 ] Allocated 33.6036 MByte for BH-tree, and 0.015625 Mbyte for top-leaves. (presently allocted 148.211 MB) [ 000010.95 ] Tree construction. (presently allocated=148.211 MB) [ 000010.96 ] Blackholes: Eliminated 0 gas particles and merged away 0 black holes. [ 000011.17 ] Tree construction done. At 'DENSITY', treewalk_run()/treewalk.c/503: Largest Allocation = 196.941 Mbyte (on task=5), Smallest = 194.484 Mbyte, Average = 195.971 Mbyte ------------------------ Allocated Memory Blocks---------------------------------------- Task Nr F Variable MBytes Cumulative Function/File/Linenumber ---------------------------------------------------------------------------------------- 5 0 0 BetaH0 0.0156 0.0156 cooling.c:568:InitCoolMemory 5 1 0 BetaHep 0.0156 0.0312 cooling.c:569:InitCoolMemory 5 2 0 AlphaHp 0.0156 0.0469 cooling.c:570:InitCoolMemory 5 3 0 AlphaHep 0.0156 0.0625 cooling.c:571:InitCoolMemory 5 4 0 Alphad 0.0156 0.0781 cooling.c:572:InitCoolMemory 5 5 0 AlphaHepp 0.0156 0.0938 cooling.c:573:InitCoolMemory 5 6 0 GammaeH0 0.0156 0.1094 cooling.c:574:InitCoolMemory 5 7 0 GammaeHe0 0.0156 0.1250 cooling.c:575:InitCoolMemory 5 8 0 GammaeHep 0.0156 0.1406 cooling.c:576:InitCoolMemory 5 9 0 Betaff 0.0156 0.1562 cooling.c:577:InitCoolMemory 5 10 0 Powerspectrum 0.1250 0.2812 gravpm.c:30:powerspectrum_a 5 11 0 Powermodes 0.0625 0.3438 gravpm.c:32:powerspectrum_a 5 12 0 NextActiveParti 1.0000 1.3438 allocate.c:23:allocate_memory 5 13 0 NextInTimeBin 1.0000 2.3438 allocate.c:25:allocate_memory 5 14 0 PrevInTimeBin 1.0000 3.3438 allocate.c:27:allocate_memory 5 15 0 P 52.0000 55.3438 allocate.c:29:allocate_memory 5 16 0 SphP 54.0000 109.3438 allocate.c:50:allocate_memory 5 17 0 BhP 5.0000 114.3438 allocate.c:54:allocate_memory 5 18 0 DomainStartList 0.0039 114.3477 domain.c:258:domain_allocate 5 19 0 DomainEndList 0.0039 114.3516 domain.c:261:domain_allocate 5 20 0 TopNodes 0.2344 114.5859 domain.c:264:domain_allocate 5 21 0 DomainNodeIndex 0.0156 114.6016 forcetree.c:1445:force_treealloc 5 22 0 Nodes_base 17.2305 131.8320 forcetree.c:1448:force_treealloc 5 23 0 Extnodes_base 14.3594 146.1914 forcetree.c:1455:force_treealloc 5 24 0 Nextnode 1.0195 147.2109 forcetree.c:1475:force_treealloc 5 25 0 Father 1.0000 148.2109 forcetree.c:1481:force_treealloc 5 26 0 Left 1.0039 149.2148 density.c.c:128:density 5 27 0 Right 1.0039 150.2188 density.c.c:129:density 5 28 0 Ngblist 16.0039 166.2227 treewalk.cc:139:ev_begin 5 29 0 DataIndexTable 4.4805 170.7031 treewalk.c:144:ev_begin 5 30 0 DataNodeList 11.9414 182.6445 treewalk.c:146:ev_begin 5 31 0 ActiveQueue 0.5039 183.1484 treewalk.c:270:treewalk_get_qu 5 32 0 PrimaryTasks 0.5078 183.6562 treewalk.c:153:ev_begin 5 33 0 currentIndexPer 0.0039 183.6602 treewalk.c:156:ev_begin 5 34 0 currentEndPerTh 0.0039 183.6641 treewalk.c:157:ev_begin 5 35 0 EvDataGet 13.2734 196.9375 treewalk.c:555:ev_get_remote ---------------------------------------------------------------------------------------- [ 000071.38 ] ngb iteration 1: need to repeat for 1924796 particles. [ 000112.29 ] ngb iteration 2: need to repeat for 473 particles. [ 000115.26 ] Converting u -> entropy, with density split sph [ 000159.13 ] iteration 000, max relative difference = 0.0154411 [ 000207.68 ] iteration 001, max relative difference = 0.00301495 ############ WARNING: EARLY TERMINATION ############# Transferring files from compute nodes to server Writing files in permanent directory /home/vat/physics/vatcosmo Final files in permanent data directory: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) ############ WARNING: EARLY TERMINATION ############# Transferring files from compute nodes to server Writing files in permanent directory /home/vat/physics/vatcosmo Final files in permanent data directory: Transferring files from compute nodes to server Writing files in permanent directory /home/vat/physics/vatcosmo Final files in permanent data directory: