Cách monitor memcached khi bị stop và ảnh hưởng table bị crash trong Drupal 7

Khi memcached ngưng hoạt động khi Available memory đã đầy. Sau đây hướng dẫn bạn kiểm tra bằng các terminal

Kiểm tra cấu hình

nano /etc/sysconfig/memcached

PORT="11211"
USER="memcached"
MAXCONN="10240"
CACHESIZE="128"
OPTIONS="-l 127.0.0.1 -U 0"

Kiểm tra trạng thái Memcached

service memcached status

Khởi động Memcached

chkconfig memcached on
service memcached start

There are several ways to retrieve stats for your memcached daemon, and to confirm all is well with how it works. Most of the time you’ll want to monitor cache HITS and MISSES, so we’ll primarily focus on this aspect.

Pick your preferred monitoring tool

Whether you don’t want to install any additional package or fancy more detailed statistics, your preferred monitoring tool will vary.

Let’s make a few assumptions first to better understand the context of this post:

• Your database server is running Ubuntu/Debian.
• You have one or several web nodes and at least one database server (alternatively you could have a primary/replica).
• memcached is installed on the remote database server.
• The database server’s IPv4 address is 203.0.113.12
• memcached is listening on its default port (11211)
• netcat is installed

Using netcat

In most Ubuntu-based distributions and Debian, netcat will be available by default. This allows to run commands such as:

$ echo stats | nc 203.0.113.12 11211
STAT pid 71
STAT uptime 6949
STAT time 1486379640
STAT version 1.4.21
STAT libevent 2.0.21-stable
STAT pointer_size 64
(snipped)
END

This gives you extensive stats (uptime, version, HITS, MISSES, connections, evictions…) about the memcached daemon. If you wish to retrieve HITS in real-time, you can build upon this command and do the following.

$ watch -td '(echo stats ; echo quit) | nc 203.0.113.12 11211 | grep get_hits '
STAT get_hits 1987

The number will then increase as soon as there’s data coming in through the memcached daemon.

Using memcstat

Installing the libmemcached-tools package will give you access to the memcstat command, which displays the operating status of a single or group of memcached servers. Usage is very simple.

$ memcstat --servers=203.0.113.12 11211 | grep get_hits
	 get_hits: 1207

Once again we’ve filtered stats out to retrieve memcached HITS only. You get the point.

Using telnet

Telnet is usually installed on Ubuntu-based distributions but not on Debian. It’s a network protocol that can help you with interacting with the memcached daemon.

Since we’re primarily interested into high-level memcached monitoring here, we’ll simply use the stats command here.

telnet 203.0.113.12 11211
Trying 203.0.113.12...
Connected to 203.0.113.12.
Escape character is '^]'.
stats
STAT pid 71
STAT uptime 10791
STAT time 1486383482
STAT version 1.4.21
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 0.356000
STAT rusage_system 0.652000
STAT curr_connections 12
STAT total_connections 804
STAT connection_structures 13
(snipped)

Using memcached-top

memcached-top was originally hosted on Google Code and now lives on Github. It’s a perl file that allows you to watch the traffic and stats in real-time. It wasn’t updated in a while but still seems to work fine.

$ perl memcache-top --instance=203.0.113.12
memcache-top v0.7	(default port: 11211, color: on, refresh: 3 seconds)

INSTANCE		      USAGE	HIT %	CONN	TIME	ITEMS	EVICT/s READ/s	WRITE/s
203.0.113.12:11211    5.3%  90.7%   10      0.6ms   867     0.0	 9207	 118.2K
AVERAGE:		      5.3%	93.4%	10      0.7ms   867     0.0     9831    114.9K

TOTAL: 3.4MB/64.0MB                 10      0.6ms   867     0.0     8999    117.2K
(ctrl-c to quit.)

Using mcstat

I find mcstat more interesting because it’s also more advanced. It wasn’t updated in a while so it might behave inconsistently as memcached evolves.

It reports on memcached daemon statistics, but in real-time, and arbitrarily returns only the stats that really matter to you when evaluating if memcached is doing its job or not. When you exit it, it immediately returns session statistics.

$ mcstat 203.0.113.12 11211
Memcache version 1.4.21
Max Memory Size: 64.0M  Pointer size: 64 bit

    time  cnct    used    gets    hits    miss    sets    evic   miss%   evic%      in     out
10:23:23    10    2.7M    1.3k    1.2k      87    6.1k       0    6.72    0.00   24.8M   11.0M

10:23:28    10    2.7M       0       0       0       0       0    0.00    0.00       7    1.1k
10:23:33    10    2.7M     209     206       3       7       0    1.44    0.00   20.0k    1.4M
10:23:38    10    2.7M     202     178      24      37       0   11.88    0.00   95.8k    1.6M
10:23:43    10    2.7M     209     175      34      43       0   16.27    0.00  181.9k    2.2M

Session Statistics:
             Statistic             Total           Session
                uptime         1h 5m 12s               20s
              libevent     2.0.21-stable                 0
           rusage_user             0.24s             0.01s
         rusage_system             0.45s             0.03s
      curr_connections                                  10
     total_connections               644                10
 connection_structures                12                 0
          reserved_fds                20                 0
               cmd_get              1.3k               620
               cmd_set              6.1k                87
             cmd_flush                 0                 0
             cmd_touch                 0                 0
              get_hits              1.2k               559
            get_misses                87                61
         delete_misses                 3                 1
           delete_hits                18                 1
           incr_misses                 0                 0
             incr_hits                 0                 0
           decr_misses                 0                 0
             decr_hits                 0                 0
            cas_misses                 0                 0
              cas_hits                 0                 0
            cas_badval                 0                 0
            touch_hits                 0                 0
          touch_misses                 0                 0
             auth_cmds                 0                 0
           auth_errors                 0                 0
            bytes_read             24.8M            297.7k
         bytes_written             11.0M              5.2M
       accepting_conns                 1                 0
   listen_disabled_num                 0                 0
               threads                 4                 0
           conn_yields                 0                 0
      hash_power_level                16                 0
            hash_bytes            524.3k                 0
     hash_is_expanding                 0                 0
          malloc_fails                 0                 0
                 bytes              2.7M             74.2k
            curr_items                             603-649
           total_items              6.1k                87
     expired_unfetched                 0                 0
     evicted_unfetched                 0                 0
             evictions                 0                 0
             reclaimed                 0                 0
     crawler_reclaimed                 0                 0
     lrutail_reflocked                 0                 0
Connection closed
Exiting.

As you can see, there are a lot of tools available for you to easily monitor memcached. Not all are created equal so it’s really up to you what’s the best tool for the job. My advice is: keep it simple until you need to dive deeper.