为什么host_statistics64（）返回的结果不一致？(Why does host_stat

2019-08-16 20:36发布

站内文章 / 后端开发
39 0 0

为什么host_statistics64（）在OS X 10.6.8（我不知道是否有其他版本有此问题）返回计数不加起来RAM总量自由，活跃，不活跃，与有线的记忆？ 为什么它丢失的页面不一致的号码？

以下输出表示不归类为自由的，有效，无效或有线十秒（每秒采样一次粗略）的页数。 

458
243
153
199
357
140
304
93
181
224

产生上述的数字的代码是： 

#include <stdio.h>
#include <mach/mach.h>
#include <mach/vm_statistics.h>
#include <sys/types.h>
#include <sys/sysctl.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char** argv) {
        struct vm_statistics64 stats;
        mach_port_t host    = mach_host_self();
        natural_t   count   = HOST_VM_INFO64_COUNT;
        natural_t   missing = 0;
        int         debug   = argc == 2 ? !strcmp(argv[1], "-v") : 0;
        kern_return_t ret;
        int           mib[2];
        long          ram;
        natural_t     pages;
        size_t        length;
        int           i;

        mib[0] = CTL_HW;
        mib[1] = HW_MEMSIZE;
        length = sizeof(long);
        sysctl(mib, 2, &ram, &length, NULL, 0);
        pages  = ram / getpagesize();

        for (i = 0; i < 10; i++) {
                if ((ret = host_statistics64(host, HOST_VM_INFO64, (host_info64_t)&stats, &count)) != KERN_SUCCESS) {
                        printf("oops\n");
                        return 1;
                }

                /* updated for 10.9 */
                missing = pages - (
                        stats.free_count     +
                        stats.active_count   +
                        stats.inactive_count +
                        stats.wire_count     +
                        stats.compressor_page_count
                );

                if (debug) {
                        printf(
                                "%11d pages (# of pages)\n"
                                "%11d free_count (# of pages free) \n"
                                "%11d active_count (# of pages active) \n"
                                "%11d inactive_count (# of pages inactive) \n"
                                "%11d wire_count (# of pages wired down) \n"
                                "%11lld zero_fill_count (# of zero fill pages) \n"
                                "%11lld reactivations (# of pages reactivated) \n"
                                "%11lld pageins (# of pageins) \n"
                                "%11lld pageouts (# of pageouts) \n"
                                "%11lld faults (# of faults) \n"
                                "%11lld cow_faults (# of copy-on-writes) \n"
                                "%11lld lookups (object cache lookups) \n"
                                "%11lld hits (object cache hits) \n"
                                "%11lld purges (# of pages purged) \n"
                                "%11d purgeable_count (# of pages purgeable) \n"
                                "%11d speculative_count (# of pages speculative (also counted in free_count)) \n"
                                "%11lld decompressions (# of pages decompressed) \n"
                                "%11lld compressions (# of pages compressed) \n"
                                "%11lld swapins (# of pages swapped in (via compression segments)) \n"
                                "%11lld swapouts (# of pages swapped out (via compression segments)) \n"
                                "%11d compressor_page_count (# of pages used by the compressed pager to hold all the compressed data) \n"
                                "%11d throttled_count (# of pages throttled) \n"
                                "%11d external_page_count (# of pages that are file-backed (non-swap)) \n"
                                "%11d internal_page_count (# of pages that are anonymous) \n"
                                "%11lld total_uncompressed_pages_in_compressor (# of pages (uncompressed) held within the compressor.) \n",
                                pages, stats.free_count, stats.active_count, stats.inactive_count,
                                stats.wire_count, stats.zero_fill_count, stats.reactivations,
                                stats.pageins, stats.pageouts, stats.faults, stats.cow_faults,
                                stats.lookups, stats.hits, stats.purges, stats.purgeable_count,
                                stats.speculative_count, stats.decompressions, stats.compressions,
                                stats.swapins, stats.swapouts, stats.compressor_page_count,
                                stats.throttled_count, stats.external_page_count,
                                stats.internal_page_count, stats.total_uncompressed_pages_in_compressor
                        );
                }

                printf("%i\n", missing);
                sleep(1);
        }

        return 0;
}

Answer 1:

TL; DR：

  • host_statistics64()得到可能花费时间，并可能产生不一致的结果不同来源的信息。
  • host_statistics64()获得通过变量与名称类似的一些信息vm_page_foo_count 。 但是，并非所有这些变量都考虑在内，如vm_page_stolen_count没有。
  • 众所周知/usr/bin/top增加了被盗的页面 有线页的数量。 这是这些页面应该计算页面时，必须考虑到的指标。

笔记

  • 我工作的一个MacOS的10.12 达尔文内核版本16.5.0 XNU-3789.51.2〜3 / RELEASE_X86_64 x86_64的 ，但所有的行为是完全地可再现。
  • 我要联系很多XNU版本的源代码，我在我的机器上使用。 它可以在这里找到： XNU-3789.51.2 。
  • 你写的程序是基本相同/usr/bin/vm_stat这仅仅是一个包装host_statistics64()和host_statistics() 该corressponding源代码可以在这里找到： system_cmds-496 / vm_stat.tproj / vm_stat.c 。

如何host_statistics64()融入XNU，它是如何工作的？

作为widley知道OS X的内核叫XNU（X NU是N OTüNIX）和“是一个混合内核组合在卡内基-梅隆大学，FreeBSD和C ++ API组件用于编写驱动程序称为由于IOKit开发马赫内核。” （ https://github.com/opensource-apple/xnu/blob/10.12/README.md ）

虚拟存储器管理（VM）是马赫的一部分因此host_statistics64()位于这里。 让我们在其实施载于细看XNU-3789.51.2 / osfmk /克恩/ host.c 。

函数签名是

kern_return_t
host_statistics64(host_t host, host_flavor_t flavor, host_info64_t info, mach_msg_type_number_t * count);

第一个相关的线

[...]
processor_t processor;
vm_statistics64_t stat;
vm_statistics64_data_t host_vm_stat;
mach_msg_type_number_t original_count;
unsigned int local_q_internal_count;
unsigned int local_q_external_count;
[...]
processor = processor_list;
stat = &PROCESSOR_DATA(processor, vm_stat);
host_vm_stat = *stat;

if (processor_count > 1) {
    simple_lock(&processor_list_lock);

    while ((processor = processor->processor_list) != NULL) {
        stat = &PROCESSOR_DATA(processor, vm_stat);

        host_vm_stat.zero_fill_count += stat->zero_fill_count;
        host_vm_stat.reactivations += stat->reactivations;
        host_vm_stat.pageins += stat->pageins;
        host_vm_stat.pageouts += stat->pageouts;
        host_vm_stat.faults += stat->faults;
        host_vm_stat.cow_faults += stat->cow_faults;
        host_vm_stat.lookups += stat->lookups;
        host_vm_stat.hits += stat->hits;
        host_vm_stat.compressions += stat->compressions;
        host_vm_stat.decompressions += stat->decompressions;
        host_vm_stat.swapins += stat->swapins;
        host_vm_stat.swapouts += stat->swapouts;
    }

    simple_unlock(&processor_list_lock);
}
[...]

我们得到host_vm_stat这类型的vm_statistics64_data_t 。 这仅仅是一个typedef struct vm_statistics64 ，你可以在看XNU-3789.51.2 / osfmk /马赫/ vm_statistics.h 。 而我们从万客隆处理器信息PROCESSOR_DATA()中定义的XNU-3789.51.2 / osfmk /克恩/ processor_data.h 。 填写host_vm_stat同时，通过我们所有的处理器通过简单地添加了相关的数字循环。

正如你所看到的，我们发现了一些众所周知的统计像zero_fill_count或compressions ，但不是所有所涵盖host_statistics64()

接下来的相关行是： 

stat = (vm_statistics64_t)info;

stat->free_count = vm_page_free_count + vm_page_speculative_count;
stat->active_count = vm_page_active_count;
[...]
stat->inactive_count = vm_page_inactive_count;
stat->wire_count = vm_page_wire_count + vm_page_throttled_count + vm_lopage_free_count;
stat->zero_fill_count = host_vm_stat.zero_fill_count;
stat->reactivations = host_vm_stat.reactivations;
stat->pageins = host_vm_stat.pageins;
stat->pageouts = host_vm_stat.pageouts;
stat->faults = host_vm_stat.faults;
stat->cow_faults = host_vm_stat.cow_faults;
stat->lookups = host_vm_stat.lookups;
stat->hits = host_vm_stat.hits;

stat->purgeable_count = vm_page_purgeable_count;
stat->purges = vm_page_purged_count;

stat->speculative_count = vm_page_speculative_count;

我们重用stat并使它我们的产量结构。 然后填写free_count与两者之和unsigned long叫vm_page_free_count和vm_page_speculative_count 。 我们收集以同样的方式在其他剩余数据（通过使用变量命名vm_page_foo_count ）或利用从统计host_vm_stat我们在上面填满。

结论我们收集不同来源的数据。 无论是从处理器的信息或变量称为vm_page_foo_count 。 这将花费时间和一些inconsitency可能最终事情的事实VM是一个非常快速和连续的过程。

让我们来看看已经提到的变量细看vm_page_foo_count 。 他们定义在XNU-3789.51.2 / osfmk / VM / vm_page.h如下： 

extern
unsigned int    vm_page_free_count; /* How many pages are free? (sum of all colors) */
extern
unsigned int    vm_page_active_count;   /* How many pages are active? */
extern
unsigned int    vm_page_inactive_count; /* How many pages are inactive? */
#if CONFIG_SECLUDED_MEMORY
extern
unsigned int    vm_page_secluded_count; /* How many pages are secluded? */
extern
unsigned int    vm_page_secluded_count_free;
extern
unsigned int    vm_page_secluded_count_inuse;
#endif /* CONFIG_SECLUDED_MEMORY */
extern
unsigned int    vm_page_cleaned_count; /* How many pages are in the clean queue? */
extern
unsigned int    vm_page_throttled_count;/* How many inactives are throttled */
extern
unsigned int    vm_page_speculative_count;  /* How many speculative pages are unclaimed? */
extern unsigned int vm_page_pageable_internal_count;
extern unsigned int vm_page_pageable_external_count;
extern
unsigned int    vm_page_xpmapped_external_count;    /* How many pages are mapped executable? */
extern
unsigned int    vm_page_external_count; /* How many pages are file-backed? */
extern
unsigned int    vm_page_internal_count; /* How many pages are anonymous? */
extern
unsigned int    vm_page_wire_count;     /* How many pages are wired? */
extern
unsigned int    vm_page_wire_count_initial; /* How many pages wired at startup */
extern
unsigned int    vm_page_free_target;    /* How many do we want free? */
extern
unsigned int    vm_page_free_min;   /* When to wakeup pageout */
extern
unsigned int    vm_page_throttle_limit; /* When to throttle new page creation */
extern
uint32_t    vm_page_creation_throttle;  /* When to throttle new page creation */
extern
unsigned int    vm_page_inactive_target;/* How many do we want inactive? */
#if CONFIG_SECLUDED_MEMORY
extern
unsigned int    vm_page_secluded_target;/* How many do we want secluded? */
#endif /* CONFIG_SECLUDED_MEMORY */
extern
unsigned int    vm_page_anonymous_min;  /* When it's ok to pre-clean */
extern
unsigned int    vm_page_inactive_min;   /* When to wakeup pageout */
extern
unsigned int    vm_page_free_reserved;  /* How many pages reserved to do pageout */
extern
unsigned int    vm_page_throttle_count; /* Count of page allocations throttled */
extern
unsigned int    vm_page_gobble_count;
extern
unsigned int    vm_page_stolen_count;   /* Count of stolen pages not acccounted in zones */
[...]
extern
unsigned int    vm_page_purgeable_count;/* How many pages are purgeable now ? */
extern
unsigned int    vm_page_purgeable_wired_count;/* How many purgeable pages are wired now ? */
extern
uint64_t    vm_page_purged_count;   /* How many pages got purged so far ? */

这是一个很多关于我们只有获得非常有限的使用数量的统计host_statistics64() 最这些统计数据都在更新XNU-3789.51.2 / osfmk / VM / vm_resident.c 。 例如该函数释放的页面的自由页面列表： 

/*
*   vm_page_release:
*
*   Return a page to the free list.
*/

void
vm_page_release(
    vm_page_t   mem,
    boolean_t   page_queues_locked)
{
    [...]
    vm_page_free_count++;
    [...]
}

非常有趣的是extern unsigned int vm_page_stolen_count; /* Count of stolen pages not acccounted in zones */ extern unsigned int vm_page_stolen_count; /* Count of stolen pages not acccounted in zones */ 。 什么是偷来的页面呢？ 好像有机制，采取了首页输出，即使它通常不会被换出一些列表。 其中一个机制是一个页面中的投机页面列表中的年龄 。 XNU-3789.51.2 / osfmk / VM / vm_page.h告诉我们 

* VM_PAGE_MAX_SPECULATIVE_AGE_Q * VM_PAGE_SPECULATIVE_Q_AGE_MS
* defines the amount of time a speculative page is normally
* allowed to live in the 'protected' state (i.e. not available
* to be stolen if vm_pageout_scan is running and looking for
* pages)...  however, if the total number of speculative pages
* in the protected state exceeds our limit (defined in vm_pageout.c)
* and there are none available in VM_PAGE_SPECULATIVE_AGED_Q, then
* vm_pageout_scan is allowed to steal pages from the protected
* bucket even if they are underage.
*
* vm_pageout_scan is also allowed to pull pages from a protected
* bin if the bin has reached the "age of consent" we've set

这的确是void vm_pageout_scan(void)即递增vm_page_stolen_count 。 你会发现在相应的源代码XNU-3789.51.2 / osfmk / VM / vm_pageout.c 。

我认为，在计算VM统计被盗的页面不考虑host_statistics64()一样。

有证据表明我是正确的

为了证明这一点，最好的办法是编译XNU用的定制版host_statistics64()手。 我有没有机会做到这一点，但会尝试很快。

幸运的是我们不感兴趣的正确VM统计唯一的。 因此，我们应该看看清楚地知道实施/usr/bin/top （不包含在XNU），这是完全可以在这里找到： 顶-108 （我只是选择了MacOS的10.12.4版本 ）。

让我们来看看顶级108 / libtop.c ，我们发现以下几点： 

static int
libtop_tsamp_update_vm_stats(libtop_tsamp_t* tsamp) {
    kern_return_t kr;
    tsamp->p_vm_stat = tsamp->vm_stat;

    mach_msg_type_number_t count = sizeof(tsamp->vm_stat) / sizeof(natural_t);
    kr = host_statistics64(libtop_port, HOST_VM_INFO64, (host_info64_t)&tsamp->vm_stat, &count);
    if (kr != KERN_SUCCESS) {
        return kr;
    }

    if (tsamp->pages_stolen > 0) {
        tsamp->vm_stat.wire_count += tsamp->pages_stolen;
    }

    [...]

    return kr;
}

tsamp是类型的libtop_tsamp_t这是在所定义的结构顶部-108 / libtop.h 。 它包含了除其他事项外vm_statistics64_data_t vm_stat和uint64_t pages_stolen 。

正如你所看到的， static int libtop_tsamp_update_vm_stats(libtop_tsamp_t* tsamp)得到tsamp->vm_stat通过充满host_statistics64()因为我们知道这一点。 然后它检查是否tsamp->pages_stolen > 0 ，并增加了它的wire_count的字段tsamp->vm_stat 。

2.结论我们不会得到这些被盗的页面数，如果我们只是用host_statistics64()中/usr/bin/vm_stat或您的示例代码！

为什么host_statistics64()实现的，因为它是什么？

老实说，我不知道。 寻呼是一个复杂的过程，因此实时观察一个具有挑战性的任务。 我们必须注意到，似乎是在其执行没有错误。 我认为我们甚至不会得到的页面的100％准确的数字，如果我们能得到访问vm_page_stolen_count 。 执行/usr/bin/top不算偷页，如果它们的数量不是很大。

一个额外的有趣的是上述函数的注释static void update_pages_stolen(libtop_tsamp_t *tsamp)是/* This is for <rdar://problem/6410098>. */ /* This is for <rdar://problem/6410098>. */ 。 打开雷达是苹果软件中的错误报告的网站，通常在评论中给出的格式分类错误。 我无法找到相关的bug; 也许它是关于缺页。

我希望这些信息能帮助你一点。 如果我设法编译XNU的最新（和定制）版本我的机器上我会让你知道。 也许这带来了有趣的见解。



Answer 2:

只注意到如果添加compressor_page_count混进去你会得到更接近实际的内存量的机器。

这是观察，而不是解释，并链接到这是适当的记录将是不错的！



文章来源: Why does host_statistics64() return inconsistent results?
标签： macos memory darwin mach
举报

收藏的人(0)

Ta的文章 更多文章
登录 后发表评论
0条评论
还没有人评论过~